Add (heavily-filtered) pixel test cq job for interactive_ui_tests
Currently, there are a number of tests which cannot be run as part of
browser_tests because they flake or fail to run on some platforms and
must be run as part of interactive_ui_tests.
These include, e.g. a number of IPH tests, but also a number of manual
tests we intend to automate as part of an ongoing test automation
project.
If a test for a secondary UI surface will not reliably run in
browser_tests, it is currently not possible to do Skia Gold visual
regression testing on that surface, as currently only browser_tests is
ever run with Skia Gold support. This is done as pixel_browser_tests, in
which the browser_tests suite is re-run with a very restrictive filter
(basically, only tests that actually take screenshots) so that only the
bare minimum of time and resources are consumed.
There was no equivalent for interactive_ui_tests, even though it should
be fairly straightforward to create one.
This CL adds a similar variation on interactive_ui_tests called
pixel_interactive_ui_tests, which invokes interactive_ui_tests with Skia
Gold support. It mimics pixel_browser_tests in the following ways:
- Shares the same filter, so that all pixel tests can be listed in the
same place and no non-pixel tests are run
- Runs on a limited set of platforms
- Does not retry failed tests
Note that any tests included in pixel_interactive_ui_tests will still
run in interactive_ui_tests with screenshotting as a no-op, just as
with pixel_browser_tests and browser_tests. This ensures that any
behavioral regressions on other platforms are still caught by CQ.
Initially, this only has the effect of re-enabling the IPH pixel test suite, which was moved to interactive_ui_tests because it would not run reliably in browser_tests (due to its sensitivity to window focus/activation.)
Bug: 1319632
Change-Id: I4fc44b404ca66fa1c82ea0873d80fedceac97fc5
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/3606835
Reviewed-by: Erik Staab <[email protected]>
Commit-Queue: Dana Fried <[email protected]>
Cr-Commit-Position: refs/heads/main@{#996361}
diff --git a/testing/buildbot/test_suite_exceptions.pyl b/testing/buildbot/test_suite_exceptions.pyl
index 2723558..45b5cdd 100644
--- a/testing/buildbot/test_suite_exceptions.pyl
+++ b/testing/buildbot/test_suite_exceptions.pyl
@@ -2581,6 +2581,19 @@
},
},
},
+ 'pixel_interactive_ui_tests': {
+ 'modifications': {
+ 'win-pixel-tester-rel': {
+ 'non_precommit_args': [
+ # Do not allow retry or it will break the bulk approval process.
+ # When retry with fail-pass pattern, the passing test will overwrite the previously
+ # seen flaky version on the trace. We can triage the image using the printed link,
+ # but it will not show on the Skia Gold search page.
+ '--test-launcher-retry-limit=0',
+ ],
+ },
+ },
+ },
'pixel_skia_gold_passthrough_test': {
'modifications': {
'Android FYI Release (Pixel 4)': {
