docs/testing/android_test_instructions.md

# Android Testing in Chromium

[TOC]

## Test Types

-   **[gtests]**: For writing unit tests in C++. Most tests are cross-platform.
-   **Browser Tests**: Built on top of gtest. Used to write integration tests.
-   **[Robolectric]**: JUnit tests that run on the host machine by emulating (or
    mocking) Android APIs.
-   **[Instrumentation Tests]**: JUnit tests that run on devices / emulators.
    - **Unit Instrumentation Tests**: Unit tests that do not require
      initializing the Content layer. These use [BaseActivityTestRule] often
      using [BlankUiTestActivity].
    - **Integration Instrumentation Tests**: Instrumentation tests that require
      initializing the Content layer to test a certain feature in the end-to-end
      flow. These typically use more specialized test rules such as
      [ContentShellActivityTestRule] or [ChromeActivityTestRule].

[gtests]: android_gtests.md
[Robolectric]: android_robolectric_tests.md
[Instrumentation Tests]: android_instrumentation_tests.md
[BaseActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:base/test/android/javatests/src/org/chromium/base/test/BaseActivityTestRule.java
[BlankUiTestActivity]: https://source.chromium.org/chromium/chromium/src/+/main:ui/android/javatests/src/org/chromium/ui/test/util/BlankUiTestActivity.java
[ContentShellActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:content/shell/android/javatests/src/org/chromium/content_shell_apk/ContentShellActivityTestRule.java
[ChromeActivityTestRule]: https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/android/javatests/src/org/chromium/chrome/test/ChromeActivityTestRule.java

## Device Setup

### Physical Device Setup

#### Root Access

Running tests requires root access, which requires using a userdebug build on
your device.

To use a userdebug build, see
[Running Builds](https://source.android.com/setup/build/running.html). Googlers
can refer to [this page](https://goto.google.com/flashdevice).

If you can't run "adb root", you will get an error when trying to install
the test APKs like "adb: error: failed to copy" and
"remote secure_mkdirs failed: Operation not permitted" (use "adb unroot" to
return adb to normal).

#### ADB Debugging

The adb executable exists within the Android SDK:

```shell
third_party/android_sdk/public/platform-tools/adb
```

In order to allow the ADB to connect to the device, you must enable USB
debugging:

* Developer options are hidden by default. To unhide them:
    *   Go to "About phone"
    *   Tap 10 times on "Build number"
    *   The "Developer options" menu will now be available.
    *   Check "USB debugging".
    *   Un-check "Verify apps over USB".

#### Screen

You **must** ensure that the screen stays on while testing: `adb shell svc power
stayon usb` Or do this manually on the device: Settings -> Developer options ->
Stay Awake.

If this option is greyed out, stay awake is probably disabled by policy. In that
case, get another device or log in with a normal, unmanaged account (because the
tests will break in exciting ways if stay awake is off).

#### Disable Verify Apps

You may see a dialog like [this
one](http://www.samsungmobileusa.com/simulators/ATT_GalaxyMega/mobile/screens/06-02_12.jpg),
which states, _Google may regularly check installed apps for potentially harmful
behavior._ This can interfere with the test runner. To disable this dialog, run:

```shell
adb shell settings put global package_verifier_enable 0
```

### Using Emulators

Running tests on emulators is the same as [on device](#Running-Tests). Refer to
[android_emulator.md](/docs/android_emulator.md) for setting up emulators.

## Building Tests

If you're adding a new test file, you'll need to explicitly add it to a gn
target. If you're adding a test to an existing file, you won't need to make gn
changes, but you may be interested in where your test winds up. In either case,
here are some guidelines for where a test belongs:

### C++

C++ test files typically belong in `<top-level directory>_unittests` (e.g.
`base_unittests` for `//base`). There are a few exceptions -- browser tests are
typically their own target (e.g. `content_browsertests` for `//content`, or
`browser_tests` for `//chrome`), and some unit test suites are broken at the
second directory rather than the top-level one.

### Java

Java test files vary a bit more widely than their C++ counterparts:

-   Instrumentation test files -- i.e., tests that will run on a device --
    typically belong in either `<top-level directory>_javatests` or `<top-level
    directory>_test_java`. Regardless, they'll wind up getting packaged into one
    of a few test APKs:
    -   `webview_instrumentation_test_apk` for anything in `//android_webview`
    -   `content_shell_test_apk` for anything in `//content` or below
    -   `chrome_public_test_apk` for most things in `//chrome`
-   JUnit or Robolectric test files -- i.e., tests that will run on the host --
    typically belong in `<top-level directory>_junit_tests` (e.g.
    `base_junit_tests` for `//base`), though here again there are cases
    (particularly in `//components`) where suites are split at the second
    directory rather than the top-level one.

Once you know what to build, just do it like you normally would build anything
else, e.g.: `ninja -C out/Release chrome_public_test_apk`

### Determining Test Target

If you do not know what target a test file belongs to, you can use
`//tools/autotest.py` to figure it out fo you:

```sh
# Builds relevant test target and then runs the test:
tools/autotest.py -C <output directory> TestClassName
```

## Running Tests

All functional tests should be runnable via the wrapper scripts generated at
build time:

```sh
<output directory>/bin/run_<target_name> [options]
```

Note that tests are sharded across all attached devices unless explicitly told
to do otherwise by `-d/--device`.

The commands used by the buildbots are printed in the logs. Look at
https://build.chromium.org/ to duplicate the same test command as a particular
builder.

### Listing Available Tests

Use `--list-tests` to list what tests are available.

```sh
# Prints out all available tests:
<output directory>/bin/run_<target_name> --list-tests

# Prints out all available tests that match a filter:
<output directory>/bin/run_<target_name> --list-tests -f "*MyFilter*"
```

### INSTALL\_FAILED\_CONTAINER\_ERROR or INSTALL\_FAILED\_INSUFFICIENT\_STORAGE

If you see this error when the test runner is attempting to deploy the test
binaries to the AVD emulator, you may need to resize your userdata partition
with the following commands:

```shell
# Resize userdata partition to be 1G
resize2fs android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img 1G

# Set filesystem parameter to continue on errors; Android doesn't like some
# things e2fsprogs does.
tune2fs -e continue android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img
```

### AdbCommandFailedError: failed to stat remote object

There's a known issue (https://crbug.com/1094062) where the unit test binaries can fail on
Android R and later: if you see this error, try rerunning on an Android version
with API level <= 29 (Android <= Q).

## Symbolizing Crashes

Crash stacks are logged and can be viewed using `adb logcat`. To symbolize the
traces, define `CHROMIUM_OUTPUT_DIR=$OUTDIR` where `$OUTDIR` is the argument you
pass to `ninja -C`, and pipe the output through
`third_party/android_platform/development/scripts/stack`. If
`$CHROMIUM_OUTPUT_DIR` is unset, the script will search `out/Debug` and
`out/Release`. For example:

```shell
# If you build with
ninja -C out/Debug chrome_public_test_apk
# You can run:
adb logcat -d | third_party/android_platform/development/scripts/stack

# If you build with
ninja -C out/android chrome_public_test_apk
# You can run:
adb logcat -d | CHROMIUM_OUTPUT_DIR=out/android third_party/android_platform/development/scripts/stack
# or
export CHROMIUM_OUTPUT_DIR=out/android
adb logcat -d | third_party/android_platform/development/scripts/stack
```

## Robolectric Tests

JUnit tests are Java unittests running on the host instead of the target device.
They are faster to run and therefore are recommended over instrumentation tests
when possible.

The JUnits tests are usually following the pattern of *target*\_junit\_tests,
for example, `content_junit_tests` and `chrome_junit_tests`.

When adding a new JUnit test, the associated `BUILD.gn` file must be updated.
For example, adding a test to `chrome_junit_tests` requires to update
`chrome/android/BUILD.gn`.

```shell
# Build the test suite.
ninja -C out/Default chrome_junit_tests

# Run the test suite.
out/Default/bin/run_chrome_junit_tests

# Run a subset of tests. You might need to pass the package name for some tests.
out/Default/bin/run_chrome_junit_tests -f "org.chromium.chrome.browser.media.*"
```

### Debugging

Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):

```shell
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger
out/Default/bin/run_chrome_junit_tests --wait-for-java-debugger  # Specify custom port via --debug-socket=9999
```

## Gtests

```shell
# Build a test suite
ninja -C out/Release content_unittests

# Run a test suite
out/Release/bin/run_content_unittests [-vv]

# Run a subset of tests and enable some "please go faster" options:
out/Release/bin/run_content_unittests --fast-local-dev -f "ByteStreamTest.*"
```

## Instrumentation Tests

In order to run instrumentation tests, you must leave your device screen ON and
UNLOCKED. Otherwise, the test will timeout trying to launch an intent.
Optionally you can disable screen lock under Settings -> Security -> Screen Lock
-> None.

Next, you need to build the app, build your tests, and then run your tests
(which will install the APK under test and the test APK automatically).

Examples:

ContentShell tests:

```shell
# Build the tests:
ninja -C out/Release content_shell_test_apk

# Run the test suite:
out/Release/bin/run_content_shell_test_apk [-vv]

# Run a subset of tests and enable some "please go faster" options:
out/Release/bin/run_content_shell_test_apk --fast-local-dev -f "*TestClass*"
```

Android WebView tests:

See [WebView's instructions](/android_webview/docs/test-instructions.md).

In order to run a subset of tests, use -f to filter based on test class/method
or -A/-E to filter using annotations.

More Filtering examples:

```shell
# Run a specific test class
out/Debug/bin/run_content_shell_test_apk -f "AddressDetectionTest.*"

# Run a specific test method
out/Debug/bin/run_content_shell_test_apk -f AddressDetectionTest#testAddressLimits

# Run a subset of tests by size (Smoke, SmallTest, MediumTest, LargeTest,
# EnormousTest)
out/Debug/bin/run_content_shell_test_apk -A Smoke

# Run a subset of tests by annotation, such as filtering by Feature
out/Debug/bin/run_content_shell_test_apk -A Feature=Navigation
```

You might want to add stars `*` to each as a regular expression, e.g.
`*`AddressDetectionTest`*`

### Debugging

Similar to [debugging apk targets](/docs/android_debugging_instructions.md#debugging-java):

```shell
out/Debug/bin/run_content_shell_test_apk --wait-for-java-debugger
```

### Deobfuscating Java Stacktraces

If running with `is_debug=false`, Java stacks from logcat need to be fixed up:

```shell
build/android/stacktrace/java_deobfuscate.py out/Release/apks/ChromePublicTest.apk.mapping < stacktrace.txt
```

Any stacks produced by test runner output will already be deobfuscated.


## Running Blink Web Tests

See [Web Tests](web_tests.md).

## Running Telemetry (Perf) Tests

See [Telemetry](https://chromium.googlesource.com/catapult/+/HEAD/telemetry/README.md).

## Running GPU tests

(e.g. the "Android Debug (Nexus 7)" bot on the chromium.gpu waterfall)

See https://www.chromium.org/developers/testing/gpu-testing for details. Use
`--browser=android-content-shell`. Examine the stdio from the test invocation on
the bots to see arguments to pass to `src/content/test/gpu/run_gpu_test.py`.