[rebaseline-cl] Generate dummy all-pass baselines when necessary
The warning about clobbering all-pass statuses (crrev.com/c/4334979) is
not useful when the affected builder/test passes expectedly (so it
generates no `actual_text` to download). In this change, `rebaseline-cl`
instead automatically generates a dummy baseline that `run_web_tests.py`
and `optimize-baselines` recognize as all-pass. This preserves subtest
-level coverage for failing platforms.
These dummy baselines should rarely be necessary, so this change's
impact on checkout size should be negligible. The dummy baselines are
self-documenting and link to the relevant web test documentation, which
this change updates.
Bug: 1324638
Test: To test the `run_web_tests.py` side:
Test: For `external/wpt/badging/badge-success.https.html`, create an
Test: abbreviated all-pass baseline for linux (host platform) and a
Test: failing baseline for win.
Test: `./run_web_tests.py -t Default external/wpt/badging` succeeds
Change-Id: I4a94fb4fda5c566ab0b5838c8ac913c6ee6cccce
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4385729
Reviewed-by: Weizhong Xia <[email protected]>
Commit-Queue: Jonathan Lee <[email protected]>
Cr-Commit-Position: refs/heads/main@{#1124532}
diff --git a/docs/testing/writing_web_tests.md b/docs/testing/writing_web_tests.md
index 295a9d56..9a80349 100644
--- a/docs/testing/writing_web_tests.md
+++ b/docs/testing/writing_web_tests.md
@@ -291,8 +291,8 @@
[rebaselining tools](./web_test_expectations.md).
Text baselines for `testharness.js` should be avoided, as having a text baseline
-associated with a `testharness.js` indicates the presence of a bug. For this
-reason, CLs that add text baselines must include a
+associated with a `testharness.js` test usually indicates the presence of a bug.
+For this reason, CLs that add text baselines must include a
[crbug.com](https://crbug.com) link for an issue tracking the removal of the
text expectations.
@@ -304,6 +304,24 @@
document Blink's current behavior, rather than using JavaScript to document
desired behavior and a text file to document current behavior.
+*** promo
+Because of [baseline fallback](./web_test_baseline_fallback.md), it may not be
+possible to [represent a platform-specific all-`PASS`
+status](https://crbug.com/1324638) by the platform baseline's absence. In such
+rare cases, `blink_tool.py rebaseline-cl` will generate a dummy baseline
+indicating to `run_web_tests.py` that all subtests are meant to pass:
+
+```
+This is a testharness.js-based test.
+All subtests passed and are omitted for brevity.
+See https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/writing_web_tests.md#Text-Test-Baselines for details.
+Harness: the test ran to completion.
+```
+
+`blink_tool.py optimize-baselines` will automatically remove these dummy
+baselines once all platforms are all-`PASS`.
+***
+
### The js-test.js Legacy Harness
*** promo
