This page is for Chromium-specific debugging tips; learning how to run gdb is out of scope.

• GDB
• Multiprocess Tricks
• Getting renderer subprocesses into gdb
• Choosing which renderers to debug
• Connecting to a running renderer
• Getting GPU subprocesses into gdb
• Getting browser_tests launched browsers into gdb
• Plugin Processes
• Single-Process mode
• Printing Chromium types
• Graphical Debugging Aid for Chromium Views
• Core files
• Running Tests
• Browser tests
• Layout tests
• UI tests
• Timeouts
• To replicate Window Manager setup on the bots
• BROWSER_WRAPPER
• Replicating Trybot Slowness
• Logging
• Seeing all LOG(foo) messages
• Seeing IPC debug messages
• Using valgrind
• Profiling
• i18n
• Breakpad
• Drag and Drop
• Tracking Down Bugs
• Isolating Regressions
• Screen recording for bug reports
• Version-specific issues
• Google Chrome
• Ubuntu Chromium
• Fedora's Chromium
• Libraries we depend on
• GTK
• Xlib
• Window Managers
• Mozilla Tips

GDB

Multiprocess Tricks

Getting renderer subprocesses into gdb

Since Chromium itself spawns the renderers, it can be tricky to grab a particular with gdb. This command does the trick:

chrome --disable-seccomp-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb --args'

The "--disable-seccomp-sandbox" flag is needed because otherwise the seccomp sandbox will kill the renderer process on startup. The "xterm" is necessary or gdb will run in the current terminal, which can get particularly confusing since it's running in the background, and if you're also running the main process in gdb, won't work at all (the two instances will fight over the terminal). To auto-start the renderers in the debugger, send the "run" command to the debugger:

chrome --disable-seccomp-sandbox --renderer-cmd-prefix='xterm -title renderer -e gdb --eval-command=run --args'

If you're using Emacs and M-x gdb, you can do

chrome "--renderer-cmd-prefix=gdb --args"

Note: using the --renderer-cmd-prefix option bypasses the zygote launcher, so the renderers won't be sandboxed. It is generally not an issue, except when you are trying to debug interactions with the sandbox. If that's what you are doing, you will need to attach your debugger to a running renderer process (see below).

You may also want to pass --disable-hang-monitor to suppress the hang monitor, which is rather annoying.

Choosing which renderers to debug

If you are starting multiple renderers then the above means that multiple gdb's start and fight over the console. Instead, you can set the prefix to point to this shell script:

#!/bin/sh

echo "**** Child $$ starting: y to debug"
read input
if [ "$input" == "y" ] ; then
  gdb --args $*
else
  $*
fi

Connecting to a running renderer

Usually ps aux | grep chrome will not give very helpful output. Try pstree -p | grep chrome to get something like

        |                      |-bash(21969)---chrome(672)-+-chrome(694)
        |                      |                           |-chrome(695)---chrome(696)-+-{chrome}(697)
        |                      |                           |                           \-{chrome}(709)
        |                      |                           |-{chrome}(675)
        |                      |                           |-{chrome}(678)
        |                      |                           |-{chrome}(679)
        |                      |                           |-{chrome}(680)
        |                      |                           |-{chrome}(681)
        |                      |                           |-{chrome}(682)
        |                      |                           |-{chrome}(684)
        |                      |                           |-{chrome}(685)
        |                      |                           |-{chrome}(705)
        |                      |                           \-{chrome}(717)

Most of those are threads. In this case the browser process would be 672 and the (sole) renderer process is 696. You can use gdb -p 696 to attach. Alternatively, you might find out the process ID from Chrome's built-in Task Manager (under the Tools menu). Right-click on the Task Manager, and enable "Process ID" in the list of columns.

Note: by default, sandboxed processes can't be attached by a debugger. To be able to do so, you will need to pass the --allow-sandbox-debugging option.

If the problem only occurs with the seccomp sandbox enabled (and the previous tricks don't help), you could try enabling core-dumps (see the Core files section). That would allow you to get a backtrace and see some local variables, though you won't be able to step through the running program.

Note: If you're interested in debugging LinuxSandboxIPC process, you can attach to 694 in the above diagram. The LinuxSandboxIPC process has the same command line flag as the browser process so that it's easy to identify it if you run pstree -pa.

Getting GPU subprocesses into gdb

Use --gpu-launcher flag instead of --renderer-cmd-prefix in the instructions for renderer above.

Getting browser_tests launched browsers into gdb

Use environment variable BROWSER_WRAPPER instead of --renderer-cmd-prefix switch in the instructions above.

Example: $ BROWSER_WRAPPER='xterm -title renderer -e gdb --eval-command=run --eval-command=quit --args' ./out/Debug/browser_tests --gtest_filter=Print

Plugin Processes

Same strategies as renderers above, but the flag is called --plugin-launcher:

chrome --plugin-launcher='xterm -e gdb --args'

Note: For now, this does not currently apply to PPAPI plugins because they currently run in the renderer process.

Single-Process mode

Depending on whether it's relevant to the problem, it's often easier to just run in "single process" mode where the renderer threads are in-process. Then you can just run gdb on the main process.

gdb --args chrome --single-process

Note that for technical reasons plugins cannot be in-process, so --single-process only puts the renderers in the browser process. The flag is still useful for debugging plugins (since it's only two processes instead of three) but you'll still need to use --plugin-launcher or another approach.

Printing Chromium types

gdb 7 lets us use Python to write pretty-printers for Chromium types. The directory tools/gdb/ contains a Python gdb scripts useful for Chromium code. There are similar scripts in WebKit (in fact, the Chromium script relies on using it with the WebKit one).

To include these pretty-printers with your gdb, put the following into ~/.gdbinit:

python
import sys
sys.path.insert(0, "/path/to/WebKit/Tools/gdb")
import webkit
sys.path.insert(0, "/path/to/chrome/src/tools/gdb")
import gdb_chrome

Pretty printers for std types shouldn't be necessary in gdb 7, but they're provided here in case you're using an older gdb. Put the following into ~/.gdbinit:

# Print a C++ string.
define ps
  print $arg0.c_str()
end

# Print a C++ wstring or wchar_t*.
define pws
  printf "\""
  set $c = (wchar_t*)$arg0
  while ( *$c )
    if ( *$c > 0x7f )
      printf "[%x]", *$c
    else
      printf "%c", *$c
    end
    set $c++
  end
  printf "\"\n"
end

More STL GDB macros

Graphical Debugging Aid for Chromium Views

The following link describes a tool that can be used on Linux, Windows and Mac under GDB.

http://code.google.com/p/chromium/wiki/GraphicalDebuggingAidChromiumViews

Core files

ulimit -c unlimited should cause all Chrome processes (run from that shell) to dump cores, with the possible exception of some sandboxed processes.

Some sandboxed subprocesses might not dump cores unless you pass the --allow-sandbox-debugging flag.

If the problem is a freeze rather than a crash, you may be able to trigger a core-dump by sending SIGABRT to the relevant process:

kill -6 [process id]

Running Tests

Many of our tests bring up windows on screen. This can be annoying (they steal your focus) and hard to debug (they receive extra events as you mouse over them). Instead, use Xvfb or Xephyr to run a nested X session to debug them, as outlined on LayoutTestsLinux.

Browser tests

By default the browser_tests forks a new browser for each test. To debug the browser side of a single test, use a command like

gdb --args out/Debug/browser_tests --single_process --gtest_filter=MyTestName

note the underscore in single_process -- this makes the test harness and browser process share the outermost process.

To debug a renderer process in this case, use the tips above about renderers.

Layout tests

See LayoutTestsLinux for some tips. In particular, note that it's possible to debug a layout test via sshing to a Linux box; you don't need anything on screen if you use Xvfb.

UI tests

UI tests are run in forked browsers. Unlike browser tests, you cannot do any single process tricks here to debug the browser. See below about BROWSER_WRAPPER.

To pass flags to the browser, use a command line like --extra-chrome-flags="--foo --bar".

Timeouts

UI tests have a confusing array of timeouts in place. (Pawel is working on reducing the number of timeouts.) To disable them while you debug, set the timeout flags to a large value:

• --test-timeout=100000000
• --ui-test-action-timeout=100000000
• --ui-test-terminate-timeout=100000000

To replicate Window Manager setup on the bots

Chromium try bots and main waterfall's bots run tests under Xvfb&icewm combination. Xvfb is an X11 server that redirects the graphical output to the memeory, and icewm is a simple window manager that is running on top of Xvfb. The behavior of icewm is markedly different when it comes to focus management and other window tasks, so test that runs fine locally may fail or be flaky on try bots. To run the tests on a local machine as on a bot, follow these steps:

Make sure you have icewm:

apt-get install icewm

Start Xvfb and icewm on a particular display:

Xvfb :6.0 & DISPLAY=:6.0 icewm &

Run your tests with graphics output redirected to that display:

DISPLAY=:6.0 ./out/Debug/browser_tests --gtest_filter="MyBrowserTest.MyActivateWindowTest"

You can look at a snapshot of the output by:

 xwd -display :6.0 -root | xwud

BROWSER_WRAPPER

You can also get the browser under a debugger by setting the BROWSER_WRAPPER environment variable. (You can use this for browser_tests too, but see above for discussion of a simpler way.)

BROWSER_WRAPPER='xterm -e gdb --args' out/Debug/browser_tests

Replicating Trybot Slowness

Trybots are pretty stressed, and can sometimes expose timing issues you can't normally reproduce locally.

You can simulate this by shutting down all but one of the CPUs (http://www.cyberciti.biz/faq/debian-rhel-centos-redhat-suse-hotplug-cpu/) and running a CPU loading tool (e.g., http://www.devin.com/lookbusy/). Now run your test. It will run slowly, but any flakiness found by the trybot should replicate locally now - and often nearly 100% of the time.

Logging

Seeing all LOG(foo) messages

Default log level hides LOG(INFO). Run with --log-level=0 and --enable-logging=stderr flags.

Newer versions of chromium with VLOG may need --v=1 too. For more VLOG tips, see the chromium-dev thread: http://groups.google.com/a/chromium.org/group/chromium-dev/browse_thread/thread/dcd0cd7752b35de6?pli=1

Seeing IPC debug messages

Run with CHROME_IPC_LOGGING=1 eg.

CHROME_IPC_LOGGING=1 out/Debug/chrome

or within gdb:

set environment CHROME_IPC_LOGGING 1

If some messages show as unknown, check if the list of IPC message headers in chrome/common/logging_chrome.cc is up-to-date. In case this file reference goes out of date, try looking for usage of macros like IPC_MESSAGE_LOG_ENABLED or IPC_MESSAGE_MACROS_LOG_ENABLED.

Using valgrind

To run valgrind on the browser and renderer processes, with our suppression file and flags:

$ cd $CHROMIUM_ROOT/src
$ tools/valgrind/valgrind.sh out/Debug/chrome

You can use valgrind on chrome and/or on the renderers e.g valgrind --smc-check=all ../sconsbuild/Debug/chrome or by passing valgrind as the argument to --render-cmd-prefix.

Beware that there are several valgrind "false positives" e.g. pickle, sqlite and some instances in webkit that are ignorable. On systems with prelink and address space randomization (e.g. Fedora), you may also see valgrind errors in libstdc++ on startup and in gnome-breakpad.

Valgrind doesn't seem to play nice with tcmalloc. To disable tcmalloc run GYP

$ cd $CHROMIUM_ROOT/src
$ build/gyp_chromium -Dlinux_use_tcmalloc=0

and rebuild.

Profiling

See https://sites.google.com/a/chromium.org/dev/developers/profiling-chromium-and-webkit and http://code.google.com/p/chromium/wiki/LinuxProfiling

i18n

We obey your system locale. Try something like:

LANG=ja_JP.UTF-8 out/Debug/chrome

If this doesn't work, make sure that the LANGUAGE, LC_ALL and LC_MESSAGE environment variables aren't set -- they have higher priority than LANG in the order listed. Alternatively, just do this:

LANGUAGE=fr out/Debug/chrome

Note that because we use GTK, some locale data comes from the system -- for example, file save boxes and whether the current language is considered RTL. Without all the language data available, Chrome will use a mixture of your system language and the language you run Chrome in.

Here's how to install the Arabic (ar) and Hebrew (he) language packs:

sudo apt-get install language-pack-ar language-pack-he language-pack-gnome-ar language-pack-gnome-he

Note that the --lang flag does not work properly for this.

On non-Debian systems, you need the gtk20.mo files. (Please update these docs with the appropriate instructions if you know what they are.)

Breakpad

See the last section of LinuxCrashDumping; you need to set a gyp variable and an environment variable for the crash dump tests to work.

Drag and Drop

If you break in a debugger during a drag, Chrome will have grabbed your mouse and keyboard so you won't be able to interact with the debugger! To work around this, run via Xephyr. Instructions for how to use Xephyr are on the LayoutTestsLinux page.

Tracking Down Bugs

Isolating Regressions

Old builds are archived here: http://build.chromium.org/buildbot/snapshots/chromium-rel-linux/

tools/bisect-builds.py in the tree automates bisecting through the archived builds. Despite a computer science education, I am still amazed how quickly binary search will find its target.

Screen recording for bug reports

sudo apt-get install gtk-recordmydesktop

Version-specific issues

Google Chrome

Google Chrome binaries don't include symbols. Googlers can read where to get symbols from the Google-internal wiki.

Ubuntu Chromium

Since we don't build the Ubuntu packages (Ubuntu does) we can't get useful backtraces from them. Direct users to https://wiki.ubuntu.com/Chromium/Debugging .

Fedora's Chromium

Like Ubuntu, but direct users to https://fedoraproject.org/wiki/TomCallaway/Chromium_Debug .

Libraries we depend on

GTK

See LinuxDebuggingGtk.

Xlib

If you're trying to track down X errors like:

The program 'chrome' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadDrawable (invalid Pixmap or Window parameter)'.

Some strategies are:

• pass --sync on the command line to make all X calls synchronous
• run chrome via xtrace
• turn on IPC debugging (see above section)

Window Managers

To test on various window managers, you can use a nested X server like Xephyr. Instructions for how to use Xephyr are on the LayoutTestsLinux page.

If you need to test something with hardware accelerated compositing (e.g., compiz), you can use Xgl (sudo apt-get install xserver-xgl). E.g.:

Xgl :1 -ac -accel glx:pbuffer -accel xv:pbuffer -screen 1024x768

Mozilla Tips

https://developer.mozilla.org/en/Debugging_Mozilla_on_Linux_FAQ