docs/testing/web_test_expectations.md

# Web Test Expectations and Baselines


The primary function of the web tests is as a regression test suite; this
means that, while we care about whether a page is being rendered correctly, we
care more about whether the page is being rendered the way we expect it to. In
other words, we look more for changes in behavior than we do for correctness.

[TOC]

All web tests have "expected results", or "baselines", which may be one of
several forms. The test may produce one or more of:

* A text file containing JavaScript log messages.
* A text rendering of the Render Tree.
* A screen capture of the rendered page as a PNG file.
* WAV files of the audio output, for WebAudio tests.

For any of these types of tests, baselines are checked into the web_tests
directory. The filename of a baseline is the same as that of the corresponding
test, but the extension is replaced with `-expected.{txt,png,wav}` (depending on
the type of test output). Baselines usually live alongside tests, with the
exception when baselines vary by platforms; read
[Web Test Baseline Fallback](web_test_baseline_fallback.md) for more
details.

Lastly, we also support the concept of "reference tests", which check that two
pages are rendered identically (pixel-by-pixel). As long as the two tests'
output match, the tests pass. For more on reference tests, see
[Writing ref tests](https://trac.webkit.org/wiki/Writing%20Reftests).

## Failing tests

When the output doesn't match, there are two potential reasons for it:

* The port is performing "correctly", but the output simply won't match the
  generic version. The usual reason for this is for things like form controls,
  which are rendered differently on each platform.
* The port is performing "incorrectly" (i.e., the test is failing).

In both cases, the convention is to check in a new baseline (aka rebaseline),
even though that file may be codifying errors. This helps us maintain test
coverage for all the other things the test is testing while we resolve the bug.

*** promo
If a test can be rebaselined, it should always be rebaselined instead of adding
lines to TestExpectations.
***

Bugs at [crbug.com](https://crbug.com) should track fixing incorrect behavior,
not lines in
[TestExpectations](../../third_party/blink/web_tests/TestExpectations). If a
test is never supposed to pass (e.g. it's testing Windows-specific behavior, so
can't ever pass on Linux/Mac), move it to the
[NeverFixTests](../../third_party/blink/web_tests/NeverFixTests) file. That
gets it out of the way of the rest of the project.

There are some cases where you can't rebaseline and, unfortunately, we don't
have a better solution than either:

1. Reverting the patch that caused the failure, or
2. Adding a line to TestExpectations and fixing the bug later.

In this case, **reverting the patch is strongly preferred**.

These are the cases where you can't rebaseline:

* The test is a reference test.
* The test gives different output in release and debug; in this case, generate a
  baseline with the release build, and mark the debug build as expected to fail.
* The test is flaky, crashes or times out.
* The test is for a feature that hasn't yet shipped on some platforms yet, but
  will shortly.

## Handling flaky tests

The
[flakiness dashboard](https://test-results.appspot.com/dashboards/flakiness_dashboard.html)
is a tool for understanding a test’s behavior over time.
Originally designed for managing flaky tests, the dashboard shows a timeline
view of the test’s behavior over time. The tool may be overwhelming at first,
but
[the documentation](https://dev.chromium.org/developers/testing/flakiness-dashboard)
should help. Once you decide that a test is truly flaky, you can suppress it
using the TestExpectations file, as described below.

We do not generally expect Chromium sheriffs to spend time trying to address
flakiness, though.

## How to rebaseline

Since baselines themselves are often platform-specific, updating baselines in
general requires fetching new test results after running the test on multiple
platforms.

### Rebaselining using try jobs

The recommended way to rebaseline for a currently-in-progress CL is to use
results from try jobs, by using the command-tool
`third_party/blink/tools/blink_tool.py rebaseline-cl`:

1. First, upload a CL.
2. Trigger try jobs by running `blink_tool.py rebaseline-cl`. This should
   trigger jobs on
   [tryserver.blink](https://ci.chromium.org/p/chromium/g/tryserver.blink/builders).
3. Wait for all try jobs to finish.
4. Run `blink_tool.py rebaseline-cl` again to fetch new baselines.
5. Commit the new baselines and upload a new patch.

This way, the new baselines can be reviewed along with the changes, which helps
the reviewer verify that the new baselines are correct. It also means that there
is no period of time when the web test results are ignored.

#### Handle bot timeouts

When a change will cause many tests to fail, the try jobs may exit early because
the number of failures exceeds the limit, or the try jobs may timeout because
more time is needed for the retries. Rebaseline based on such results are not
suggested. The solution is to temporarily increase the number of shards in
[test_suite_exceptions.pyl](https://source.chromium.org/chromium/chromium/src/+/main:testing/buildbot/test_suite_exceptions.pyl) in your CL.
Change the values back to its original value before sending the CL to CQ.

#### Options

The tests which `blink_tool.py rebaseline-cl` tries to download new baselines for
depends on its arguments.

* By default, it tries to download all baselines for tests that failed in the
  try jobs.
* If you pass `--only-changed-tests`, then only tests modified in the CL will be
  considered.
* You can also explicitly pass a list of test names, and then just those tests
  will be rebaselined.
* If some of the try jobs failed to run, and you wish to continue rebaselining
  assuming that there are no platform-specific results for those platforms,
  you can add the flag `--fill-missing`.
* By default, it finds the try jobs by looking at the latest patchset. If you
  have finished try jobs that are associated with an earlier patchset and you
  want to use them instead of scheduling new try jobs, you can add the flag
  `--patchset=n` to specify the patchset. This is very useful when the CL has
  'trivial' patchsets that are created e.g. by editing the CL descrpition.

### Rebaseline script in results.html

Web test results.html linked from bot job result page provides an alternative
way to rebaseline tests for a particular platform.

* In the bot job result page, find the web test results.html link and click it.
* Choose "Rebaseline script" from the dropdown list after "Test shown ... in format".
* Click "Copy report" (or manually copy part of the script for the tests you want
  to rebaseline).
* In local console, change directory into `third_party/blink/web_tests/platform/<platform>`.
* Paste.
* Add files into git and commit.

The generated command includes `blink_tool.py optimize-baselines <tests>` which
removes redundant baselines. However, the optimization doesn't work for
flag-specific baselines for now, so the rebaseline script may create redundant
baselines for flag-specific results. We prefer local manual rebaselining (see
below) for flag-specific rebaselines when possible.

### Local manual rebaselining

```bash
third_party/blink/tools/run_web_tests.py --reset-results foo/bar/test.html
```

If there are current expectation files for `web_tests/foo/bar/test.html`,
the above command will overwrite the current baselines at their original
locations with the actual results. The current baseline means the `-expected.*`
file used to compare the actual result when the test is run locally, i.e. the
first file found in the [baseline search path](https://cs.chromium.org/search/?q=port/base.py+baseline_search_path).

If there are no current baselines, the above command will create new baselines
in the platform-independent directory, e.g.
`web_tests/foo/bar/test-expected.{txt,png}`.

When you rebaseline a test, make sure your commit description explains why the
test is being re-baselined.