Got the bot started using the following recipe:

  ssh chromeosbuild28
  cd /b/slave
  make restart

Bot started immediately. I've seen this same problem before on other bots where it didn't respond even to 'force build' commands. But then when I restart the bot it shows all the times everybody hit 'force build'.

The same bot just failed trying to sync svn, perhaps there is a connectivity issue?


Syncing projects:  54% ( 6/11) 
Syncing projects:  63% ( 7/11) 
Syncing projects:  72% ( 8/11) 

________ running 'svn update /b/build/scripts/private/data/reliability --ignore-externals' in '/b'
At revision 96205.
Syncing projects:  81% ( 9/11) 

________ running 'svn update /b/build/scripts/command_wrapper/bin --ignore-externals' in '/b'
svn: Can't connect to host 'svn.chromium.org': Connection timed out
Error: svn update /b/build/scripts/command_wrapper/bin --ignore-externals in /b returned 1
program finished with exit code 1
elapsedTime=342.291587

(No comment was entered for this change.)

(No comment was entered for this change.)

So it looks like we have multiple things going on here.

First the bug says  git clone failed. I see a reference to a URL that I cannot get to nor do I recognize. David James can you comment on that? 

The second part about the bot not restarting properly. I am looking at that now. It looks like it was't started by the init scripts since I do not see any failed start attempt in the logs just the reboot and the restart by david james. 

Thirdly we are now seeing gclient sync errors which are SVN and have nothing to do with git.

A connection problem hypothesis is actually not a bad one at all as it would affect all three of these areas.  Have we seen sporadic behavior on other bots like this? If it is a connection issue I would suspect it is more the room's out going connection rather than on this particular bot. 

(No comment was entered for this change.)

This system is clearly sick. I am going to stop the build process on it and do a few things to it. 

[ 4076.377592]  [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
[ 4076.377603] INFO: task sync:14281 blocked for more than 120 seconds.
[ 4076.377879] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4076.378243] sync          D 0000000000000001     0 14281  14280 0x00000008
[ 4076.378247]  ffff8802c059fea8 0000000000000086 0000000000015d00 0000000000015d00
[ 4076.378252]  ffff880300f99aa8 ffff8802c059ffd8 0000000000015d00 ffff880300f996f0
[ 4076.378256]  0000000000015d00 ffff8802c059ffd8 0000000000015d00 ffff880300f99aa8
[ 4076.378260] Call Trace:
[ 4076.378264]  [<ffffffff81540417>] __mutex_lock_slowpath+0xf7/0x180
[ 4076.378268]  [<ffffffff815402fb>] mutex_lock+0x2b/0x50
[ 4076.378272]  [<ffffffff8116ba0f>] sync_filesystems+0x1f/0x130
[ 4076.378275]  [<ffffffff8116bb77>] sys_sync+0x17/0x40
[ 4076.378279]  [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
[ 4196.180669] INFO: task dpkg:6683 blocked for more than 120 seconds.
[ 4196.180958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4196.212304] dpkg          D 00000000ffffffff     0  6683   6657 0x00000000
[ 4196.212311]  ffff880214433d38 0000000000000086 0000000000015d00 0000000000015d00
[ 4196.212317]  ffff88031a49c888 ffff880214433fd8 0000000000015d00 ffff88031a49c4d0
[ 4196.212321]  0000000000015d00 ffff880214433fd8 0000000000015d00 ffff88031a49c888
[ 4196.212326] Call Trace:
[ 4196.212336]  [<ffffffff8153fc5d>] schedule_timeout+0x22d/0x300
[ 4196.212342]  [<ffffffff8105dcf2>] ? enqueue_entity+0x132/0x1b0
[ 4196.212346]  [<ffffffff8105ddcb>] ? enqueue_task_fair+0x5b/0xa0
[ 4196.212350]  [<ffffffff8153f87b>] wait_for_common+0xdb/0x180
[ 4196.212353]  [<ffffffff8105cb2b>] ? try_to_wake_up+0x2fb/0x480
[ 4196.212357]  [<ffffffff8105ccb0>] ? default_wake_function+0x0/0x20
[ 4196.212361]  [<ffffffff8153f9dd>] wait_for_completion+0x1d/0x20
[ 4196.212366]  [<ffffffff811670e7>] sync_inodes_sb+0x87/0xb0
[ 4196.212370]  [<ffffffff8116b9e2>] __sync_filesystem+0x82/0x90
[ 4196.212374]  [<ffffffff8116bac9>] sync_filesystems+0xd9/0x130
[ 4196.212378]  [<ffffffff8116bb81>] sys_sync+0x21/0x40
[ 4196.212384]  [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
[ 4196.212395] INFO: task sync:14281 blocked for more than 120 seconds.
[ 4196.228528] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4196.259843] sync          D 0000000000000001     0 14281  14280 0x00000008
[ 4196.259850]  ffff8802c059fea8 0000000000000086 0000000000015d00 0000000000015d00
[ 4196.259856]  ffff880300f99aa8 ffff8802c059ffd8 0000000000015d00 ffff880300f996f0
[ 4196.259860]  0000000000015d00 ffff8802c059ffd8 0000000000015d00 ffff880300f99aa8
[ 4196.259864] Call Trace:
[ 4196.259870]  [<ffffffff81540417>] __mutex_lock_slowpath+0xf7/0x180
[ 4196.259874]  [<ffffffff815402fb>] mutex_lock+0x2b/0x50
[ 4196.259878]  [<ffffffff8116ba0f>] sync_filesystems+0x1f/0x130
[ 4196.259881]  [<ffffffff8116bb77>] sys_sync+0x17/0x40
[ 4196.259886]  [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b

I upgraded the system including a kernel upgrade, That is what has generally fixed these issue that i've seen. I will watch this bot but I anticipate this problem is gone. The slave also started properly. David James informed me that SVN is having some issues so I believe that accounts for all problems we've seen except the 403 error. Which right now is considered "Sporadic" I am downgrading this for now but I will be keeping an eye on this system. 

Saw same HTTP 403 on x86-pineview full:
http://build.chromium.org/p/chromiumos/builders/x86%20pineview%20full/builds/276

Subsequent builds were OK.  nsylvain suggests that cloning should be done from git.chromium.org instead of gerrit.chromium.org.

Filed https://code.google.com/p/chromium-os/issues/detail?id=20049 for updating the gerrit.chromium.org references to git.chromium.org.

Again.

http://chromegw.corp.google.com/i/chromiumos/builders/x86%20generic%20full/builds/590/steps/Clear%20and%20Clone%20chromite/logs/stdio

rm -rf chromite ; sleep 10 ;/usr/bin/git clone http://gerrit.chromium.org/gerrit/p/chromiumos/chromite;cd chromite;/usr/bin/git checkout master
 in dir /b/build/slave/x86-generic-full/build (timeout 1200 secs)
 watching logfiles {}
 argv: rm -rf chromite ; sleep 10 ;/usr/bin/git clone http://gerrit.chromium.org/gerrit/p/chromiumos/chromite;cd chromite;/usr/bin/git checkout master
 environment:
  CHROME_HEADLESS=1
  DISPLAY=:0.0
  HOME=/home/chrome-bot
  LANG=en_US.UTF-8
  LOGNAME=chrome-bot
  PAGER=cat
  PATH=/b/build/../depot_tools:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
  PWD=/b/build/slave/x86-generic-full/build
  PYTHONPATH=/b/build/third_party/buildbot_7_12:/b/build/third_party/twisted_8_1:/b/build/site_config:/b/build/scripts:/b/build/scripts/release:/b/build/third_party:/b/build_internal/site_config:/b/build_internal/symsrc:.
  SHELL=/bin/bash
  USER=chrome-bot
 closing stdin
 using PTY: True
Cloning into chromite...
error: The requested URL returned error: 403 while accessing http://gerrit.chromium.org/gerrit/p/chromiumos/chromite/info/refs

fatal: HTTP request failed
cd: 1: can't cd to chromite
fatal: Not a git repository (or any of the parent directories): .git
program finished with exit code 128
elapsedTime=10.392768

This is marked as a TreeCloser. Please confirm that it should be a TreeCloser or remove the label. Please reassign as necessary.

This should already be fixed by sosa's change to clone from git.chromium.org

(No comment was entered for this change.)

If this should already be fixed then lets mark it as fixed, the last update was a month ago before Nick's ping. 

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)