docs/linux/debugging.md

# Tips for debugging on Linux

This page is for Chromium-specific debugging tips; learning how to run gdb is
out of scope.

[TOC]

## Symbolized stack trace

The sandbox can interfere with the internal symbolizer. Use `--no-sandbox` (but
keep this temporary) or an external symbolizer (see
`tools/valgrind/asan/asan_symbolize.py`).

Generally, do not use `--no-sandbox` on waterfall bots, sandbox testing is
needed. Talk to security@chromium.org.

## GDB

*** promo
GDB-7.7 is required in order to debug Chrome on Linux.
***

Any prior version will fail to resolve symbols or segfault.

### Setup

#### Build setup

In your build set the GN build variable `symbol_level = 2` for interactive
debugging. (`symbol_level = 1` only provides backtrace information). And while
release-mode debugging is possible, things will be much easier in a debug build.
Set your build args with `gn args out/<your_dir>` (substituting your build
directory), and set:

```
is_debug = true
symbol_level = 2
```

#### GDB setup

The Chrome build requires some GDB configuration for it to be able to find
source files. See [gdbinit](../gdbinit.md) to configure GDB. There is a similar
process for [LLDB](../lldbinit.md).

### Basic browser process debugging

    gdb -tui -ex=r --args out/Debug/chrome --disable-seccomp-sandbox \
        http://google.com

### Allowing attaching to foreign processes

On distributions that use the
[Yama LSM](https://www.kernel.org/doc/Documentation/security/Yama.txt) (that
includes Ubuntu and Chrome OS), process A can attach to process B only if A is
an ancestor of B.

You will probably want to disable this feature by using

    echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

If you don't you'll get an error message such as "Could not attach to process".

Note that you'll also probably want to use `--no-sandbox`, as explained below.

### Multiprocess Tricks

#### Getting renderer subprocesses into gdb

Since Chromium itself spawns the renderers, it can be tricky to grab a
particular with gdb. This command does the trick:

```
chrome --no-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb --args'
```

The `--no-sandbox` flag is needed because otherwise the seccomp sandbox will
kill the renderer process on startup, or the setuid sandbox will prevent xterm's
execution.  The "xterm" is necessary or gdb will run in the current terminal,
which can get particularly confusing since it's running in the background, and
if you're also running the main process in gdb, won't work at all (the two
instances will fight over the terminal). To auto-start the renderers in the
debugger, send the "run" command to the debugger:

    chrome --no-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb \
        -ex run --args'

If you're using Emacs and `M-x gdb`, you can do

    chrome "--renderer-cmd-prefix=gdb --args"

*** note
Note: using the `--renderer-cmd-prefix` option bypasses the zygote launcher, so
the renderers won't be sandboxed. It is generally not an issue, except when you
are trying to debug interactions with the sandbox. If that's what you are doing,
you will need to attach your debugger to a running renderer process (see below).
***

You may also want to pass `--disable-hang-monitor` to suppress the hang monitor,
which is rather annoying.

You can also use `--renderer-startup-dialog` and attach to the process in order
to debug the renderer code. Go to
https://www.chromium.org/blink/getting-started-with-blink-debugging for more
information on how this can be done.

For utilities you can use `--utility-startup-dialog` to have all utilities
prompt, or `--utility-startup-dialog=data_decoder.mojom.DataDecoderService`
to debug only a particular service type.

#### Choosing which renderers to debug

If you are starting multiple renderers then the above means that multiple gdb's
start and fight over the console. Instead, you can set the prefix to point to
this shell script:

```sh
#!/bin/sh

echo "**** Child $$ starting: y to debug"
read input
if [ "$input" = "y" ] ; then
  gdb --args $*
else
  $*
fi
```

#### Choosing renderer to debug by URL

In most cases you'll want to debug the renderer which is loading a particular
site. If you want a script which will automatically debug the renderer which has
visited a given target URL and continue all other renderers, you can use the
following:

```sh
./third_party/blink/tools/debug_renderer out/Default/content_shell https://example.domain/path
```

The script also supports specifying a different URL than the navigation URL.
This is useful when the renderer you want to debug is not the top frame but one
of the subframes on the page. For example, you could debug a particular subframe
on a page with:

```sh
./third_party/blink/tools/debug_renderer -d https://subframe.url/path out/Default/content_shell https://example.domain/path
```

However, if you need more fine-grained control over which renderers to debug
you can run chrome or content_shell directly with the
`--wait-for-debugger-on-navigation` flag which will pause each renderer at the
point of navigation (when the URL is known).

This will result in a series of lines such as the following in the output:
```
...:content_switches_internal.cc(119)] Renderer url="https://example.domain/path" (PID) paused waiting for debugger to attach. Send SIGUSR1 to unpause.
```

You can signal the renderers you aren't interested in to continue running with:
```sh
kill -s SIGUSR1 <pid>
```

And debug the renderer you are interested in debugging with:
```sh
gdb -p <pid>
```

#### Debugging run_web_tests.py renderers

The `debug_renderer` script can also be used to debug the renderer running
a web test. To do so, simply call `run_{web,wpt}_tests.py` from `debug_renderer`
with all of the standard arguments for `run_{web,wpt}_tests.py`. For example:

```sh
./third_party/blink/tools/debug_renderer ./third_party/blink/tools/run_web_tests.py [run_web_test args]
```

#### Selective breakpoints

When debugging both the browser and renderer process, you might want to have
separate set of breakpoints to hit. You can use gdb's command files to
accomplish this by putting breakpoints in separate files and instructing gdb to
load them.

```
gdb -x ~/debug/browser --args chrome --no-sandbox --disable-hang-monitor \
    --renderer-cmd-prefix='xterm -title renderer -e gdb -x ~/debug/renderer \
    --args '
```

Also, instead of running gdb, you can use the script above, which let's you
select which renderer process to debug. Note: you might need to use the full
path to the script and avoid `$HOME` or `~/.`

#### Connecting to a running renderer

Usually `ps aux | grep chrome` will not give very helpful output. Try
`pstree -p | grep chrome` to get something like

```
        |                      |-bash(21969)---chrome(672)-+-chrome(694)
        |                      |                           |-chrome(695)---chrome(696)-+-{chrome}(697)
        |                      |                           |                           \-{chrome}(709)
        |                      |                           |-{chrome}(675)
        |                      |                           |-{chrome}(678)
        |                      |                           |-{chrome}(679)
        |                      |                           |-{chrome}(680)
        |                      |                           |-{chrome}(681)
        |                      |                           |-{chrome}(682)
        |                      |                           |-{chrome}(684)
        |                      |                           |-{chrome}(685)
        |                      |                           |-{chrome}(705)
        |                      |                           \-{chrome}(717)
```

Most of those are threads. In this case the browser process would be 672 and the
(sole) renderer process is 696. You can use `gdb -p 696` to attach.
Alternatively, you might find out the process ID from Chrome's built-in Task
Manager (under the Tools menu). Right-click on the Task Manager, and enable
"Process ID" in the list of columns.

Note: by default, sandboxed processes can't be attached by a debugger. To be
able to do so, you will need to pass the `--allow-sandbox-debugging` option.

If the problem only occurs with the seccomp sandbox enabled (and the previous
tricks don't help), you could try enabling core-dumps (see the **Core files**
section).  That would allow you to get a backtrace and see some local variables,
though you won't be able to step through the running program.

Note: If you're interested in debugging LinuxSandboxIPC process, you can attach
to 694 in the above diagram. The LinuxSandboxIPC process has the same command
line flag as the browser process so that it's easy to identify it if you run
`pstree -pa`.

#### Getting GPU subprocesses into gdb

Use `--gpu-launcher` flag instead of `--renderer-cmd-prefix` in the instructions
for renderer above.

#### Getting `browser_tests` launched browsers into gdb

Use environment variable `BROWSER_WRAPPER` instead of `--renderer-cmd-prefix`
switch in the instructions above.

Example:

```shell
BROWSER_WRAPPER='xterm -title renderer -e gdb --eval-command=run \
    --eval-command=quit --args' out/Debug/browser_tests --gtest_filter=Print
```

#### Plugin Processes

Same strategies as renderers above, but the flag is called `--plugin-launcher`:

    chrome --plugin-launcher='xterm -e gdb --args'

*** note
Note: For now, this does not currently apply to PPAPI plugins because they
currently run in the renderer process.
***

#### Single-Process mode

Depending on whether it's relevant to the problem, it's often easier to just run
in "single process" mode where the renderer threads are in-process. Then you can
just run gdb on the main process.

    gdb --args chrome --single-process

Currently, the `--disable-gpu` flag is also required, as there are known crashes
that occur under TextureImageTransportSurface without it. The crash described in
https://crbug.com/361689 can also sometimes occur, but that crash can be
continued from without harm.

Note that for technical reasons plugins cannot be in-process, so
`--single-process` only puts the renderers in the browser process. The flag is
still useful for debugging plugins (since it's only two processes instead of
three) but you'll still need to use `--plugin-launcher` or another approach.

### Printing Chromium types

gdb 7 lets us use Python to write pretty-printers for Chromium types. See
[gdbinit](../gdbinit.md)
to enable pretty-printing of Chromium types.  This will import Blink
pretty-printers as well.

Pretty printers for std types shouldn't be necessary in gdb 7, but they're
provided here in case you're using an older gdb. Put the following into
`~/.gdbinit`:

```
# Print a C++ string.
define ps
  print $arg0.c_str()
end

# Print a C++ wstring or wchar_t*.
define pws
  printf "\""
  set $c = (wchar_t*)$arg0
  while ( *$c )
    if ( *$c > 0x7f )
      printf "[%x]", *$c
    else
      printf "%c", *$c
    end
    set $c++
  end
  printf "\"\n"
end
```

[More STL GDB macros](http://www.yolinux.com/TUTORIALS/src/dbinit_stl_views-1.01.txt)

### JsDbg -- visualize data structures in the browser

JsDbg is a debugger plugin to display various Chrome data structures in a
browser window, such as the accessibility tree, layout object tree, DOM tree,
and others.
[Installation instructions are here](https://github.com/MicrosoftEdge/JsDbg),
and see [here](https://github.com/MicrosoftEdge/JsDbg/blob/master/docs/FEATURES.md)
for screenshots and an introduction.

For Googlers, please see [go/jsdbg](https://goto.google.com/jsdbg) for
installation instructions.

### Time travel debugging with rr

You can use [rr](https://rr-project.org) for time travel debugging, so you
can also step or execute backwards. This works by first recording a trace
and then debugging based on that.

For Googlers, if you have a remote cloud machine, please follow this
[instruction](https://engdoc.corp.google.com/eng/doc/devguide/debugging/rr.md#setting-up-rr)
to set up the machine in order to use the rr tool.

You need an up-to-date version of rr, since rr is frequently updated to support
new parts of the Linux system call API surface that Chromium uses. If you have
any issues with the latest release version, try compiling rr
[from source](https://github.com/rr-debugger/rr/wiki/Building-And-Installing).

Once installed, you can use it like this:
```
rr record out/Debug/content_shell --single-process
rr replay
(rr) c
(rr) break blink::NGBlockNode::Layout
(rr) rc # reverse-continue to the last Layout call
(rr) jsdbg # run JsDbg as described above to find the interesting object
(rr) watch -l box_->frame_rect_.size_.width_.value_
(rr) rc # reverse-continue to the last time the width was changed
(rr) rn # reverse-next to the previous line
(rr) reverse-fin # run to where this function was called from
```

You can debug multi-process chrome using `rr -f [PID]`
for processes `fork()`ed from a [zygote process](zygote.md) without exec,
which includes renderer processes,
or `rr -p [PID]` for other processes.
To find the process id you can either run `rr ps` after recording, or for
renderer processes use `--vmodule=render_frame_impl=1` which will log a
message on navigations. Example:

```
$ rr record out/Debug/content_shell --disable-hang-monitor --vmodule=render_frame_impl=1 https://www.google.com/
rr: Saving execution to trace directory `...'.
...
[128515:128515:0320/164124.768687:VERBOSE1:render_frame_impl.cc(4244)] Committed provisional load: https://www.google.com/
```

From the log message we can see that the site was loaded into process 128515
and can set a breakpoint for when that process is forked.

```
rr replay -f 128515
```

If you want to call debugging functions from gdb that use `LOG()`,
then those functions need to disable the printing of timestamps using
[`SetLogItems`](https://source.chromium.org/search?q=SetLogItems&sq=&ss=chromium%2Fchromium%2Fsrc).
See `LayoutObject::ShowLayoutObject()` for an example of this, and
[issue 2829](https://github.com/rr-debugger/rr/issues/2829) for why it is needed.

If rr doesn't work correctly, the rr developers are generally quite responsive
to [bug reports](https://github.com/rr-debugger/rr/issues),
especially ones that have enough information so that
they don't have to build Chromium.

See Also:

* [The Chromium Chronicle #13: Time-Travel Debugging with RR](https://developer.chrome.com/blog/chromium-chronicle-13/)
* [@davidbaron demo using rr](https://twitter.com/davidbaron/status/1473761042278887433)
* [@davidbaron demo using pernosco](https://twitter.com/davidbaron/status/1475836824409022469)
(Googlers: see [go/pernosco](https://goto.google.com/pernosco))