forgejo/tests/e2e/README.md

# End to end tests

Thank you for your effort to provide good software tests for Forgejo.
Please also read the general testing instructions in the
[Forgejo contributor documentation](https://forgejo.org/docs/next/contributor/testing/)
and make sure to also check the
[Playwright documentation](https://playwright.dev/docs/intro)
for further information.

This file is meant to provide specific information for the integration tests
as well as some tips and tricks you should know.

Feel free to extend this file with more instructions if you feel like you have something to share!


## How to run the tests?

Before running any tests, please ensure you perform a clean frontend build:

```
make clean frontend
```

Whenever you modify frontend code (i.e. JavaScript and CSS files),
you need to create a new frontend build.

For tests that require interactive Git repos,
you also need to ensure a Forgejo binary is ready to be used by Git hooks.
For this, you additionally need to run

~~~
make TAGS="sqlite sqlite_unlock_notify" backend
~~~

### Install dependencies

Browsertesting is performed by playwright.
You need certain system libraries and playwright will download required browsers.
Playwright takes care of this when you run:

```
npx playwright install-deps
```

> **Note**
> On some operating systems, the installation of missing libraries can complicate testing certain browsers.
> It is often not necessary to test with all browsers locally.
> Choosing either Firefox or Chromium is fine.


### Run all tests

If you want to run the full test suite, you can use

```
make test-e2e-sqlite
```

### Interactive testing

We recommend that you use interactive testing for the development.
After you performed the required builds,
you should use one shell to start the debugserver (and leave it running):

```
make test-e2e-debugserver
```

It allows you to explore the test data in your local browser,
and playwright to perform tests on it.

> **Note**
> The modifications persist while the debugserver is running.
> If you modified things, it might be useful to restart it to get back to a fresh state.
> While writing playwright tests, you either
> need to ensure they are resilient against repeated runs
> (e.g. when only creating new content),
> or that they restore the initial state for the next browser run.

#### With the playwright UI:

Playwright ships with an integrated UI mode which allows you to
run individual tests and to debug them by seeing detailed traces of what playwright does.
Launch it with:

```
npx playwright test --ui
```

#### Running individual tests

```
npx playwright test actions.test.e2e.ts:9
```

First, specify the complete test filename,
and after the colon you can put the linenumber where the test is defined.


#### With VSCodium or VSCode

To debug a test, you can also use "Playwright Test" for
[VScodium](https://open-vsx.org/extension/ms-playwright/playwright)
or [VSCode](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright).


### Run all tests via local act_runner

If you have a [forgejo runner](https://code.forgejo.org/forgejo/runner/),
you can use it to run the test jobs:

```
forgejo-runner exec -W .forgejo/workflows/testing.yml -j test-e2e
```

Note that the CI workflow has some logic to run tests based on changed files only.
This might conflict with your local setup and not run all the desired tests
because it might only look at file changes in your latest commit.

### Run e2e tests with another database

This approach is not currently used,
neither in the CI/CD nor by core contributors on their local machines.
It is still documented for the sake of completeness:
You can also perform e2e tests using MariaDB/MySQL or PostgreSQL if you want.

Setup a MySQL database inside docker
```
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(Ctrl-c to stop the database)
```
Start tests based on the database container
```
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test?multiStatements=true TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-e2e-mysql
```

Setup a pgsql database inside docker
```
docker run -e POSTGRES_DB=test -e POSTGRES_PASSWORD=password -p 5432:5432 --rm --name pgsql postgres:latest #(Ctrl-c to stop the database)
```
Start tests based on the database container
```
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-e2e-pgsql
```

### Running individual tests

Example command to run `example.test.e2e.ts` test file:

> **Note**
> Unlike integration tests, this filtering is at the file level, not function

For SQLite:

```
make test-e2e-sqlite#example
```

### Visual testing

> **Warning**
> This is not currently used by most Forgejo contributors.
> Your help to improve the situation and allow for visual testing is appreciated.

Although the main goal of e2e is assertion testing, we have added a framework for visual regression testing. If you are working on front-end features, please use the following:
 - Check out `main`, `make clean frontend`, and run e2e tests with `VISUAL_TEST=1` to generate outputs. This will initially fail, as no screenshots exist. You can run the e2e tests again to assert that it passes.
 - Check out your branch, `make clean frontend`, and run e2e tests with `VISUAL_TEST=1`. You should be able to assert that your front-end changes don't break any other tests unintentionally.

`VISUAL_TEST=1` will create screenshots in tests/e2e/test-snapshots. The test will fail the first time this is enabled (until we get visual test image persistence figured out), because it will be testing against an empty screenshot folder.

`ACCEPT_VISUAL=1` will overwrite the snapshot images with new images.


## Tips and tricks

If you know noteworthy tests that can act as an inspiration for new tests,
please add some details here.

### Understanding and waiting for page loads

[Waiting for a load state](https://playwright.dev/docs/api/class-frame#frame-wait-for-load-state)
sound like a convenient way to ensure the page was loaded,
but it only works once and consecutive calls to it
(e.g. after clicking a button which should reload a page)
return immediately without waiting for *another* load event.

If you match something which is on both the old and the new page,
you might succeed before the page was reloaded,
although the code using a `waitForLoadState` might intuitively suggest
the page was changed before.

Interacting with the page before the reload
(e.g. by opening a dropdown)
might then race and result in flaky tests,
depending on the speed of the hardware running the test.

A possible way to test that an interaction worked is by checking for a known change first.
For example:

- you submit a form and you want to check that the content persisted
- checking for the content directly would succeed even without a page reload
- check for a success message first (will wait until it appears), then verify the content

Alternatively, if you know the backend request that will be made before the reload,
you can explicitly wait for it:

~~~js
const submitted = page.waitForResponse('/my/backend/post/request');
await page.locator('button').first().click(); // perform your interaction
await submitted;
~~~

If the page redirects to another URL,
you can alternatively use:

~~~js
await page.waitForURL('**/target.html');
~~~

### Only sign in if necessary

Signing in takes time and is actually executed step-by-step.
If your test does not rely on a user account, skip this step.

~~~js
test('For anyone', async ({page}) => {
  await page.goto('/somepage');
~~~

If you need a user account, you can use something like:

~~~js
import {test, login_user, login} from './utils_e2e.ts';

test.beforeAll(async ({browser}, workerInfo) => {
  await login_user(browser, workerInfo, 'user2'); // or another user
});

test('For signed users only', async ({browser}, workerInfo) => {
  const page = await login({browser}, workerInfo);
~~~

### Run tests very selectively

Browser testing can take some time.
If you want to iterate fast,
save your time and only run very selected tests.
Use only one browser.

### Skip Safari if it doesn't work

Many contributors have issues getting Safari (webkit)
and especially Safari Mobile to work.

At the top of your test function, you can use:

~~~javascript
test.skip(workerInfo.project.name === 'Mobile Safari', 'Unable to get tests working on Safari Mobile.');
~~~

### Don't forget the formatting.

When writing tests without modifying other frontend code,
it is easy to forget that the JavaScript test files also need formatting.

Run `make lint-frontend-fix`.

### Define new repos

Take a look at `declare_repos_test.go` to see how to add your repositories.
Feel free to improve the logic used there if you need more advanced functionality,
it is a simplified version of the code used in the integration tests.

### Accessibility testing

If you can, perform automated accessibility testing using
[AxeCore](https://github.com/dequelabs/axe-core-npm/blob/develop/packages/playwright/README.md).

Take a look at `shared/forms.ts` and some other places for inspiration.

### List related files coverage

To speed up the CI pipelines and avoid running expensive tests too often,
only a selection of tests is run by default, based on the changed files.

At the top of each playwright test file,
list the files or file patterns that are covered by your test.
Often, these are files that you modified for your feature or bugfix,
or that you looked at (and might still have open in your IDE),
because your fix depends on their behaviour.

#### Which files to watch?

The set of files your test "watches" depends on the kind of test you write.
If you only test for the presence of an element and do no accessibility or placement checks,
you won't detect broken visual appearance and there is little reason to watch CSS files.

However, if your test also checks that an element is correctly positioned
(e.g. that it does not overflow the page),
or has accessibility properties (includes colour contrast),
also list stylesheets that define the behaviour your test depends on.

Watching the place that generate the selectors you use
(typically templates, but can also be JavaScript)
is a must, to ensure that someone modifying the markup notices that your selectors fail
(e.g. because an id or class was renamed).

If you are unsure about the exact set of files, feel free to ask other contributors.

#### How to specify the patterns?

You put filenames and patterns as blocks between two `// @watch` comments.
An example that watches changes on (in order)
a single file,
a full recursive subfolder,
two files with a shorthand pattern,
and a set of files with a certain ending:

~~~
// @watch start
// templates/webhook/shared-settings.tmpl
// templates/repo/settings/**
// web_src/css/{form,repo}.css
// web_src/css/modules/*.css
// @watch end
~~~

The patterns are evaluated on a "first-match" basis.
Under the hood, [gobwas/glob](https://github.com/gobwas/glob) is used.

## Grouped retry for interactions

Sometimes, it can be necessary to retry certain interactions together.
Consider the following procedure:

1. click to open a dropdown
2. interact with content in the dropdown

When for some reason the dropdown does not open,
for example because of it taking time to initialize after page load,
the click will succeed,
but the depending interaction won't,
although playwright repeatedly tries to find the content.

You can [group statements using toPass]()https://playwright.dev/docs/test-assertions#expecttopass).
This code retries the dropdown click until the second item is found.

~~~js
await expect(async () => {
  await page.locator('.dropdown').click();
  await page.locator('.dropdown .item').first().click();
}).toPass();
~~~