From http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex%20canary/builds/885

Looking at the machine I noticed it was stuck on the archive_build step.

$ ps -eo pid,etime,cputime,cmd | grep -E 'bash ./cros_gen|dump_syms' | grep -v grep


15140    02:15:27 015695    02:15:22 00:00:05 /bin/bash ./cros_generate_breakpad_symbols --board=x86-alex
16636    02:08:33 00:00:00 sudo dump_syms /build/x86-alex/usr/lib/libgmock_main.so.0.0.0 /build/x86-alex/usr/lib/debug/usr/lib
16637    02:08:33 00:00:00 [dump_syms] <defunct>

Looks like that dump_syms process has been running for a long time, but it's hung in sudo rather than in the actual process it was running.

$ strace -p 16636
Process 16636 attached - interrupt to quit
select(6, [5], [], NULL, NULL

$ sudo lsof -p 16636
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
sudo    16636 root  cwd    DIR                9,1     4096 20644171 /b/cbuild/x86-alex-canary-master/chroot/home/chrome-bot/trunk/src/scripts
sudo    16636 root  rtd    DIR                9,1     4096 15442363 /b/cbuild/x86-alex-canary-master/chroot
sudo    16636 root  txt    REG                9,1   168368 21540052 /b/cbuild/x86-alex-canary-master/chroot/usr/bin/sudo
sudo    16636 root  mem    REG                9,1    47432 21523413 /b/cbuild/x86-alex-canary-master/chroot/lib64/libnss_files-2.10.1.so
sudo    16636 root  mem    REG                9,1    43384 21523433 /b/cbuild/x86-alex-canary-master/chroot/lib64/libnss_nis-2.10.1.so
sudo    16636 root  mem    REG                9,1    88880 21523437 /b/cbuild/x86-alex-canary-master/chroot/lib64/libnsl-2.10.1.so
sudo    16636 root  mem    REG                9,1    31432 21523409 /b/cbuild/x86-alex-canary-master/chroot/lib64/libnss_compat-2.10.1.so
sudo    16636 root  mem    REG                9,1  1399984 21523440 /b/cbuild/x86-alex-canary-master/chroot/lib64/libc-2.10.1.so
sudo    16636 root  mem    REG                9,1    96632 21523304 /b/cbuild/x86-alex-canary-master/chroot/lib64/libz.so.1.2.5
sudo    16636 root  mem    REG                9,1    14512 21523446 /b/cbuild/x86-alex-canary-master/chroot/lib64/libdl-2.10.1.so
sudo    16636 root  mem    REG                9,1    55712 21523459 /b/cbuild/x86-alex-canary-master/chroot/lib64/libpam.so.0.83.0
sudo    16636 root  mem    REG                9,1    10464 21523481 /b/cbuild/x86-alex-canary-master/chroot/lib64/libutil-2.10.1.so
sudo    16636 root  mem    REG                9,1   123168 21523496 /b/cbuild/x86-alex-canary-master/chroot/lib64/ld-2.10.1.so
sudo    16636 root  mem    REG                9,1  1575552 21728251 /b/cbuild/x86-alex-canary-master/chroot/usr/lib64/locale/locale-archive
sudo    16636 root    0r  FIFO                0,8      0t0    32497 pipe
sudo    16636 root    1w   REG                9,1     1062 25019071 /b/cbuild/x86-alex-canary-master/chroot/tmp/sym.bJff
sudo    16636 root    2w   REG                9,1     1339 18726942 /b/cbuild/x86-alex-canary-master/chroot/tmp/err.13To
sudo    16636 root    3r   REG                9,1     1521 21531563 /b/cbuild/x86-alex-canary-master/chroot/etc/passwd
sudo    16636 root    4r   REG                9,1      788 21531541 /b/cbuild/x86-alex-canary-master/chroot/etc/group
sudo    16636 root    5u  unix 0xffff8803229b8000      0t0  8072223 socket


I killed the offending sudo process that was hung and the build continued fine. It's strange that it looked like the sudo process was hung and not the actual dump_syms process.

See also  Bug 19386  which affected the same build because cbuildbot didn't handle the hang correctly.