docs/testing/chromeos_debugging_tips.md

# Chrome OS Debugging Instructions
Chrome on Chrome OS is tested using a handful of frameworks, each of which
you'll find running on Chrome's CQ and waterfalls. If you're investigating
failures in these tests, below are some tips for debugging and identifying the
cause.

*** note

This doc outlines tests running in true Chrome OS environments (ie: on virtual
machines or real devices). [linux-chromeos] tests, on the other hand, can be
debugged like any other linux test.
***

## Tast

[Tast] is Chrome OS's integration testing framework. Since Chrome itself is
instrumental to the Chrome OS system, it's equally important that we run some
of these integration tests on Chrome's waterfalls. If you find one of these
tests failing (likely in the `chrome_all_tast_tests` step), you can:

- **Inspect the failed test's log snippet**: There should be a log snippet for
each failed test in the `Test Results` tab in the build UI. eg: For this
[failed build], clicking on the `policy.IncognitoModeAvailability` expands to
include stack traces and error messages.

- **View browser & system logs**: A common cause of failure on Chrome's builders
are browser crashes. When this happens, each test's log snippets will simply
contain warnings like "[Chrome probably crashed]". To debug these crashes,
expand the list of attached artifacts for the test by clicking the `Artifacts`
link under the failed test in the `Test Results` tab. There you'll find an
extended log for the test under `log.txt`. Additionally, you can find system
logs included in that list. To find a system log for a particular test, match
the timestamps printed in the test's log with the timestamps present in the
system log filename. For instance, the previous `example.ChromeFixture` failure
matches the [chrome/chrome_20210920-051805] browser log, which contains the
culprit Chrome crash and backtrace.

- **Symbolizing a browser crash dump**: See [below](#symbolizing-a-crash-dump).

### Disabling a test

If you are a Chrome Sheriff, please read the sheriff documentation
[here](http://go/chrome-sheriff-tast) before disabling any tests.

Tast tests are run under both Chrome's builders and CrOS's builders. They can be
disabled either completely (in all builders), or in Chrome's builders alone. The
latter should be used only for changes which are not expected to occur on CrOS's
builders.

- **Disabling in all builders**: If you have a full CrOS checkout, you can add
the `informational` [attribute] to the test's definition. (You may be able to
bypass the need for a full CrOS checkout by using the `Edit code` button in
codesearch UI, but this flow is unverified.) This can take time (ie: many hours)
to land and propagate onto Chrome's builders. So if you need the test disabled
ASAP, consult the next option.
- **Disabling in only Chrome's builders**: You can add the test to the list of
disabled tests for the step's GN target. For example, to disable a test in the
`chrome_all_tast_tests` step, add it to [this list]. **Note**: If the test is
failing consistently, and you only disable it here, it will likely start to fail
in the next [Chrome uprev] on CrOS's builders, which can lead to further
problems down the road. So please make sure you pursue the first option as well
in that case.

In both cases, please make sure a bug is filed for the test, and route it to
the appropriate owners.

### Symbolizing a crash dump

If a test fails due to a browser crash, there should be a Minidump crash report
present in the test's isolated out under the prefix `crashes/chrome...`. These
reports aren't very useful by themselves, but with a few commands you can
symbolize the report locally to get insight into what conditions caused Chrome
to crash.

If you are running a locally compiled [Simple Chrome] binary on a device or VM,
you can can build `minidump_stackwalk` and download the
`/home/chronos/crash/chrome*.dmp` file.
```
autoninja -C out/Release minidump_stackwalk dump_syms

rsync -r -e "ssh -p 9222" root@localhost:/home/chronos/crash /tmp
```

For a crash on a bot, download both the task's input files (this provides the
symbols and the symbolizing tools) as well as the task's output results (this
provides the crash reports). See the commands listed under the *Reproducing the
task locally* section on the task page. For example, to download them for
[this task](https://chrome-swarming.appspot.com/task?id=5cc272e0a839b311), `cd`
into a tmp directory and run:
```
cipd install "infra/tools/luci/cas/\${platform}" -root bar
./bar/cas login
./bar/cas download -cas-instance projects/chrome-swarming/instances/default_instance -digest 1ad29e201e4ae7e3056a8b17935edbcd62fb54befdfeba221f2e82e54f150c86/812 -dir foo

cipd install "infra/tools/luci/swarming/\${platform}" -root bar
./bar/swarming login
./bar/swarming collect -S chrome-swarming.appspot.com -output-dir=foo 5cc272e0a839b311
```

Generate the breakpad symbols by pointing the `generate_breakpad_symbols.py` script to
your local binary, or the downloaded input build dir:
```
cd foo
vpython3 components/crash/content/tools/generate_breakpad_symbols.py --symbols-dir symbols --build-dir out/Release/ --binary out/Release/chrome --platform chromeos
```

That will generate the symbols in the `symbols/` dir. Then to symbolize a Chrome
crash report (either in the tasks's output, or the `/tmp/crash` dir):
```
./out/Release/minidump_stackwalk 5cc272e0a839b311/crashes/chrome.20220816.214251.44917.24579.dmp symbols/
```


### Running a test locally

To run a Tast test the same way it's ran on Chrome's builders:

- Decide which Chrome OS device type or VM to test on.

- Build Chrome via the [Simple Chrome] workflow for that board.

- Deploy your Chrome to the device via the [deploy_chrome.py] tool.

- Finally, run the Tast test on the device via the `cros_run_test` tool under
  `//third_party/chromite/bin/`. eg:
  `cros_run_test --device $DEVICE_IP --tast login.Chrome`. See [here] for more
  info on cros_run_test.

## Telemetry

>TODO: Add instructions for debugging telemetry failures.

## GTest

>TODO: Add instructions for debugging GTest failures.

## Rerunning these tests locally

>TODO: Add instructions for rerunning these tests locally.


[linux-chromeos]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/chromeos_build_instructions.md
[Tast]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/README.md
[failed build]: https://ci.chromium.org/ui/p/chromium/builders/ci/chromeos-kevin-rel/37300/test-results
[Chrome probably crashed]: https://luci-milo.appspot.com/ui/inv/build-8835572137562508161/test-results?q=example.ChromeFixture
[chrome/chrome_20210920-051805]: https://luci-milo.appspot.com/ui/artifact/raw/invocations/task-chromium-swarm.appspot.com-561bed66572a9411/artifacts/chrome%2Fchrome_20210920-051805
[attribute]: https://chromium.googlesource.com/chromiumos/platform/tast/+/HEAD/docs/test_attributes.md
[this list]: https://codesearch.chromium.org/chromium/src/chromeos/tast_control.gni
[Chrome uprev]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/chrome_commit_pipeline.md#the-chrome-os-commit-pipeline-for-chrome-changes
[Simple Chrome]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md
[deploy_chrome.py]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/simple_chrome_workflow.md#Deploying-Chrome-to-the-device
[here]: https://chromium.googlesource.com/chromiumos/docs/+/HEAD/cros_vm.md#in-simple-chrome