Bumping severity, it's affecting more builders now.

A minimal improvement to the test which would provide more information would be to actually go out and check whether or not the browser is in fact running, in order to differentiate a browser crash from a browser hang.

I think in this case it's this:

----
Unable to access a Google product
If you've been redirected to this page from a particular product, it means that your access to this product has been suspended. Read on for more information.

Your access to this Google product has been suspended because of a perceived violation of either the Google Terms of Service or product-specific Terms of Service. For specific product guidelines, please visit the homepage of each Google product you're interested in for a link to its Terms of Service.
-----

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)

If we hit Gaia as in comment @2, we will time out because AutomationProvider::OnLoginWebuiReady will never be called.

"WaitForInitialLoads failed" seems to show up all over, even in tests where we use the local auth server. login_CryptohomeUnmounted also uses local auth server too.

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex-binary/builds/1372/steps/VMTest/logs/stdio for another example of this

I've added debug logging to chrome code to track this failure (r101411). However chrome didn't uprev last night, so it doesn't have my patch. Waiting..

another similar looking one:
failed: suite_Smoke/desktopui_WindowManagerFocusNewWindows
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-zgb_he%20canary/builds/110/steps/VMTest/logs/stdio

I've verified that this happens because webui login never got online, so the automation could not initialize.
Note in the logs that I see OnInitialTabLoadsComplete and OnNetworkLibraryInit but no OnLoginWebuiReady as expected.
Xiyuan, suggestions?

https://sandbox.google.com/storage/?arg=chromeos-image-archive/x86-generic-pre-flight-queue/0.16.1059.0-a1-b915#chromeos-image-archive%2Fx86-generic-pre-flight-queue%2F0.16.1059.0-a1-b915

chrome logs: ------
[2461:2461:47652129:VERBOSE2:automation_provider.cc(215)] OnInitialTabLoadsComplete
[2461:2461:47815057:VERBOSE2:automation_provider.cc(222)] OnNetworkLibraryInit
[2461:2461:47980069:WARNING:user_cros_settings_provider.cc(409)] No metrics policy set will revert to checking consent file which is on.
[2461:2461:48003760:WARNING:proxy_config_service_impl.cc(662)] Error retrieving proxy setting from device
[2461:2461:48495465:INFO:CONSOLE(3327)] "Opening extension: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html, opt_email=", source: chrome://oobe/ (3327)
[2461:2461:48618450:INFO:CONSOLE(158)] "#### main.html start", source: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html?hl=en-US (158)
[2461:2461:48618835:INFO:CONSOLE(48)] "### Authenticator.initialize", source: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html?hl=en-US (48)
[2461:2461:48633095:INFO:CONSOLE(100)] "#### Authenticator.onPageLoad: navigator.onLine = false", source: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html?hl=en-US (100)
[2461:2497:56622395:WARNING:audio_mixer_alsa.cc(207)] Attach to card default failed: No such file or directory
[2461:2461:92833750:WARNING:passive_log_collector.cc(311)] Released a reference to nonexistent source.
[2461:2461:92833781:WARNING:passive_log_collector.cc(311)] Released a reference to nonexistent source.
[2461:2461:92834727:INFO:CONSOLE(3327)] "Opening extension: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html, opt_email=", source: chrome://oobe/ (3327)
[2461:2461:92900913:INFO:CONSOLE(158)] "#### main.html start", source: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html?hl=en-US&test_email=performancetestaccount%40gmail.com&test_password=perfsmurf (158)
[2461:2461:92901459:INFO:CONSOLE(48)] "### Authenticator.initialize", source: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html?hl=en-US&test_email=performancetestaccount%40gmail.com&test_password=perfsmurf (48)
[2461:2461:92917408:INFO:CONSOLE(100)] "#### Authenticator.onPageLoad: navigator.onLine = false", source: chrome-extension://mfffpogegjflfpflabcdkioaeobkgjik/main.html?hl=en-US&test_email=performancetestaccount%40gmail.com&test_password=p
erfsmurf (100)
~                                                                     

desktopui_UrlFetch.not-live.DEBUG ---------
09/19 13:52:32 INFO |logging_ma:0560| chrome/test/ui/ui_test.cc:133: Failure
09/19 13:52:32 INFO |logging_ma:0560| Value of: launcher_->InitializeConnection(DefaultLaunchState(), wait_for_initial_loads_)
09/19 13:52:32 ERROR|logging_ma:0560| [2242:2242:0919/135232:92816442:ERROR:proxy_launcher.cc(108)] WaitForInitialLoads failed.
09/19 13:52:32 INFO |logging_ma:0560|   Actual: false
09/19 13:52:32 ERROR|logging_ma:0560| [2242:2242:0919/135232:92816545:ERROR:proxy_launcher.cc(558)] Failed to ConnectToRunningBrowser
09/19 13:52:32 INFO |logging_ma:0560| Expected: true
09/19 13:52:48 DEBUG|miniFakeDn:0086| Response: clients4.google.com. -> 127.0.0.1
09/19 13:52:48 DEBUG|miniFakeDn:0086| Response: clients4.google.com. -> 127.0.0.1
09/19 13:52:48 DEBUG|miniFakeDn:0086| Response: clients4.google.com. -> 127.0.0.1
09/19 13:53:17 INFO |logging_ma:0560| chrome/test/pyautolib/pyautolib.cc:338: Failure
09/19 13:53:17 INFO |logging_ma:0560| Value of: automation()->SendJSONRequest(request, timeout, &response)
09/19 13:53:17 INFO |logging_ma:0560|   Actual: false

Thanks nirnimesh to get all the logs.

I'll check why chrome has the wrong online status.

Updating title.

Reproduction: http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex%20canary/builds/1025/steps/VMTest/logs/stdio

Think zel's CL http://codereview.chromium.org/7969009/ should fix this as well.

(No comment was entered for this change.)

Reproduced:
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex-binary/builds/1448

(No comment was entered for this change.)

http://build.chromium.org/p/chromiumos/builders/x86%20pineview%20full/builds/414/steps/VMTest/logs/stdio

#20: Attaching VMTest log file 

pineviewVMTest414.txt 27.8 KB   View   Download

Jason and I found a case where flimflam timed out when trying to resolve www.google.com during portal detection and report chrome PORTAL state. Portal is not treated as online in chrome right now and thus chrome is not loading Gaia and eventually automation timed out.

x86-generic PFQ #956 failed with this symptom:
   http://build.chromium.org/p/chromiumos/builders/x86%20generic%20PFQ/builds/956

See VMTest log.

Another x86-generic PFQ fail:
http://build.chromium.org/p/chromiumos/builders/x86%20generic%20PFQ/builds/958/steps/VMTest/logs/stdio

Majority of bots are all red with this error - 9/22 10 AM EST.

Tree is being left closed to resolve this.

as of 4AM bot-time (Pacific?) we had rolled in chrome 889.0, which is at revision 102207.  People were saying yesterday that fixes had landed in chrome, and we needed to wait for them.  I don't know if the bots kliegs refers to had this version of chrome.  The external bots look mostly green, so he must be talking about internal bots that I can't see because I'm not on corp right now :-)

However, if the builds in question used CHrome 888, and not 889...it is worth kicking them again.

My fixes to a related issue were in revision 102209.

buildspec hasn't landed yet. Still seeing failures:
http://build.chromium.org/p/chromiumos/builders/x86%20generic%20PFQ/builds/963/steps/VMTest/logs/stdio

eblake created http://codereview.chromium.org/7994006/ and zelidrag's fix(es) should land "shortly" in ChromiumOS builds.

As of 4:15 PM EST chrome build 889.2 has been rev'd and should have the change.  http://chrome-master2.mtv.corp.google.com/official_builds/16.0.889.2/summary.html reports it as being at revision 102305 (change was 102209).  

Will watch future builds for confirmation

Unfortunately this is still happening:
Failing build: http://chromeos-botmaster.mtv.corp.google.com:8026/builders/lumpy-binary/builds/192

cbuildbot output: http://chromeos-botmaster.mtv.corp.google.com:8026/builders/lumpy-binary/builds/192/steps/cbuildbot/logs/stdio

vmtest failure: http://chromeos-botmaster.mtv.corp.google.com:8026/builders/lumpy-binary/builds/192/steps/VMTest/logs/stdio

And another one: http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex-binary/builds/1473

Chrome logs have:
[2567:2567:55177211:VERBOSE2:automation_provider.cc(215)] OnInitialTabLoadsComplete
[2567:2567:55234673:VERBOSE2:automation_provider.cc(222)] OnNetworkLibraryInit

But no: "OnLoginWebuiReady" as expected for the automation to connect and proceed.

flimflam stuck in portal state is tracked in  issue 20726 .

I'll investigate why chrome still thinks it's offline with zel's fix. :(

And another:
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/TOT%20Pre-Flight%20Queue/builds/5696

Commit: 70ebe9d77ad247e1d370c3f21f20b1ea96832e6b
 Email: jglasgow@chromium.org

flimflam: portal: fix timeout computation, use curl_multi_perform

Fix the timeout computation when tv_nsec overflows.  This might cause
incorrect computation of expiration times.

Use curl_multi_perform instead of curl_multi_socket_action with
CURL_SOCKET_TIMEOUT because the former causes curl to handle timed out
requests properly, whereas curl_multi_socket_action does not complete
the timedout requests.

BUG=chromium-os:20323
TEST=while ssh root@jglinux1.cam -p 9222 -o StrictHostKeyChecking=no /usr/local/lib/flimflam/test/list-services | grep online ; do sudo pkill kvm ; ./bin/cros_start_vm  ; sleep 15; done

Change-Id: Icac84ba874888480e1b32a48cc2b75ce93e4b33e
Reviewed-on: http://gerrit.chromium.org/gerrit/8137
Commit-Ready: Jason Glasgow <jglasgow@chromium.org>
Reviewed-by: Jason Glasgow <jglasgow@chromium.org>
Tested-by: Jason Glasgow <jglasgow@chromium.org>

M	src/portal.c

I am no longer to repro it in my VM after zel's change. 
Added more logging from chromeos net notifier in r102446.

Repeated on /builders/x86-alex%20canary/builds/1044/steps/VMTest/logs/stdio

Loks like it repeated on builders/lumpy-binary/builds/214/steps/VMTest/logs/stdio:

17:37:17 INFO | 	START	login_CryptohomeMounted	login_CryptohomeMounted	timestamp=1316911035	localtime=Sep 24 17:37:15	
17:37:17 INFO | Bundling /build/lumpy/usr/local/autotest/client/site_tests/login_CryptohomeMounted into test-login_CryptohomeMounted.tar.bz2
17:38:12 INFO | 		ERROR	login_CryptohomeMounted	login_CryptohomeMounted	timestamp=1316911089	localtime=Sep 24 17:38:09	Unhandled JSONInterfaceError: Automation call {'username': 'performancetestaccount@gmail.com', 'password': 'perfsmurf', 'command': 'Login'} received empty response.  Perhaps the browser crashed.
17:38:12 INFO | 	END ERROR	login_CryptohomeMounted	login_CryptohomeMounted	timestamp=1316911089	localtime=Sep 24 17:38:09	
17:38:12 INFO | 	START	login_CryptohomeIncognitoUnmounted	login_CryptohomeIncognitoUnmounted	timestamp=1316911090	localtime=Sep 24 17:38:10	

status 1.7 KB   View   Download

chrome_20110924-173718 3.7 KB   View   Download

chrome_20110924-173719 9.5 KB   View   Download

Another one from zgb-binary:

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-zgb%20canary/builds/840/steps/VMTest/logs/stdio

3:10:20 INFO | 	END GOOD	login_CryptohomeIncognitoMounted	login_CryptohomeIncognitoMounted	timestamp=1316981420	localtime=Sep 25 13:10:20	
13:10:22 INFO | 	START	login_CryptohomeMounted	login_CryptohomeMounted	timestamp=1316981422	localtime=Sep 25 13:10:22	
13:10:23 INFO | Bundling /build/x86-zgb/usr/local/autotest/client/site_tests/login_CryptohomeMounted into test-login_CryptohomeMounted.tar.bz2
13:12:03 INFO | 		ERROR	login_CryptohomeMounted	login_CryptohomeMounted	timestamp=1316981523	localtime=Sep 25 13:12:03	Unhandled JSONInterfaceError: Automation call {'username': 'performancetestaccount@gmail.com', 'password': 'perfsmurf', 'command': 'Login'} received empty response.  Perhaps the browser crashed.
13:12:03 INFO | 	END ERROR	login_CryptohomeMounted	login_CryptohomeMounted	timestamp=1316981523	localtime=Sep 25 13:12:03	


========= ERROR FILE /tmp/cbuildbot95a5oZ/test_harness/testUpdateKeepStateful/4_verify/suite_Smoke/login_CryptohomeMounted/debug/login_CryptohomeMounted.ERROR FOR TEST suite_Smoke/login_CryptohomeMounted ==============
09/25 13:10:24 ERROR|  warnings:0029| /usr/local/lib/python2.6/site-packages/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
09/25 13:10:24 ERROR|  warnings:0029|   super(Connection, self).__init__(*args, **kwargs)
09/25 13:11:14 ERROR|logging_ma:0560| [10567:10567:0925/131114:209365746:ERROR:proxy_launcher.cc(109)] WaitForInitialLoads failed.
09/25 13:11:14 ERROR|logging_ma:0560| [10567:10567:0925/131114:209365853:ERROR:proxy_launcher.cc(562)] Failed to ConnectToRunningBrowser

Similar one for x86-zgb_he:

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-zgb_he%20canary/builds/146/steps/VMTest/logs/stdio


========== ERROR FILE /tmp/cbuildbotgNIRuP/test_harness/testUpdateWipeStateful/2_verify/suite_Smoke/login_CryptohomeUnmounted/debug/login_CryptohomeUnmounted.ERROR FOR TEST suite_Smoke/login_CryptohomeUnmounted ==============
09/25 12:41:14 ERROR|  warnings:0029| /usr/local/lib/python2.6/site-packages/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
09/25 12:41:14 ERROR|  warnings:0029|   super(Connection, self).__init__(*args, **kwargs)
09/25 12:42:11 ERROR|logging_ma:0560| [13145:13145:0925/124211:743658830:ERROR:proxy_launcher.cc(109)] WaitForInitialLoads failed.
09/25 12:42:11 ERROR|logging_ma:0560| [13145:13145:0925/124211:743658929:ERROR:proxy_launcher.cc(562)] Failed to ConnectToRunningBrowser

#38 is something else instead of "WaitForInitialLoads failed"

#39 and #40 are real cases. And in both cases, we now have "navigator.onLine = true" in the log after zel's change. However, "OnLoginWebuiReady" is not there. This means we did not successfully load Gaia/mock Gaia.

I looked into 39 and 40, and in both cases I never see an attempt to fetch /accounts/ServiceLogin

I see successful DNS resolutions of accounts.google.com, though.  In successful runs, I don't see anyone try to go to accounts.google.com

2 questions:
1) What's the URL that webui login tries to load from accounts.google.com?
2) What makes the webui login screen decide between using www.google.com/accounts/ServiceLogin and accounts.google.com/whatever?

extension uses https://www.google.com/accounts/ServiceLogin?service=chromeoslogin by default.

For cases in #39 and #40, chrome log shows the extension's attempts to load Gaia (search for "Loading GAIA frame"). However, we did not get the "clearOldAttemps" call from Gaia/mock Gaia.

mock gaia never received a request for /accounts/ServiceLogin in those cases.  I see other requests from chrome, for /searchdomaincheck and some help content.

Looked at #40 case and compared one success test with the failed login_CryptohomeUnmounted log.

When we succeed, we hit /accounts/ServeceLogin pretty fast. When we fail, we do a lot more stuff. The suspicious log lines are the lines where we try to hit "ocsp.thawte.com".

How does the https work with mock Gaia? Is it possible that chrome got some certifcate error and stopped fetching mock Gaia?

We inject a fake root cert into the user's NSS database, as though they'd added a custom CA certificate.

It's very possible that one of the recent SSL hardening things is getting in the way, as we _are_ specifically faking out www.google.com :-/

I wonder if actually chaining a cert off of the root cert I install and using that for the server will work.  Or, maybe we'll need to respond on whatever endpoint Chrome is hitting to fake it out.  Can you figure out a way to gather more info from whatever Chrome-side code might be doing this?  I'll investigate the cert-faking-side tomorrow.

Again twice last night.  Here's the log for one:  http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex-binary/builds/1510

I have a pending CL http://codereview.chromium.org/8037035/ that will dump our the frame navigation error number. This would give us more hint. That CL will also retry loading Gaia if we are online and not in a portal.

Looks like another instance:
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-zgb-binary/builds/1398/steps/VMTest/logs/stdio


11:18:26 INFO | 	START	security_ProfilePermissions.login	security_ProfilePermissions.login	timestamp=1317147506	localtime=Sep 27 11:18:26	
11:18:26 INFO | Bundling /build/x86-zgb/usr/local/autotest/client/site_tests/security_ProfilePermissions into test-security_ProfilePermissions.tar.bz2
11:19:19 INFO | 		ERROR	security_ProfilePermissions.login	security_ProfilePermissions.login	timestamp=1317147560	localtime=Sep 27 11:19:20	Unhandled JSONInterfaceError: Automation call {'username': 'performancetestaccount@gmail.com', 'password': 'perfsmurf', 'command': 'Login'} received empty response.  Perhaps the browser crashed.
11:19:20 INFO | 	END ERROR	security_ProfilePermissions.login	security_ProfilePermissions.login	timestamp=1317147560	localtime=Sep 27 11:19:20	

@47 and @49 are not related to this issue. I could not find "WaitForInitialLoads failed" in VMTest log. And taking a closer look at debug and chrome log, automation successfully initiated login. The failures are something else.

47 and 49 are 20887; xiyuan and I looked at the logs together

Another likely instance:
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/lumpy%20canary/builds/93/steps/VMTest/logs/stdio

========== ERROR FILE /tmp/cbuildbotLl2bgb/test_harness/testUpdateWipeStateful/2_verify/suite_Smoke/desktopui_WindowManagerFocusNewWindows/debug/desktopui_WindowManagerFocusNewWindows.ERROR FOR TEST suite_Smoke/desktopui_WindowManagerFocusNewWindows ==============
09/27 13:03:34 ERROR|  warnings:0029| /usr/local/lib/python2.6/site-packages/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
09/27 13:03:34 ERROR|  warnings:0029|   super(Connection, self).__init__(*args, **kwargs)
09/27 13:05:11 ERROR|logging_ma:0560| [3374:3374:0927/130511:209366483:ERROR:proxy_launcher.cc(109)] WaitForInitialLoads failed.
09/27 13:05:11 ERROR|logging_ma:0560| [3374:3374:0927/130511:209366567:ERROR:proxy_launcher.cc(562)] Failed to ConnectToRunningBrowser

Again here:
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/lumpy-binary/builds/237/steps/VMTest/logs/stdio

========== ERROR FILE /tmp/cbuildbotfvkwVT/test_harness/testUpdateWipeStateful/2_verify/suite_Smoke/desktopui_WindowManagerFocusNewWindows/debug/desktopui_WindowManagerFocusNewWindows.ERROR FOR TEST suite_Smoke/desktopui_WindowManagerFocusNewWindows ==============
09/27 14:44:06 ERROR|  warnings:0029| /usr/local/lib/python2.6/site-packages/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
09/27 14:44:06 ERROR|  warnings:0029|   super(Connection, self).__init__(*args, **kwargs)
09/27 14:45:00 ERROR|logging_ma:0560| [3326:3326:0927/144500:159434726:ERROR:proxy_launcher.cc(109)] WaitForInitialLoads failed.
09/27 14:45:00 ERROR|logging_ma:0560| [3326:3326:0927/144500:159434828:ERROR:proxy_launcher.cc(562)] Failed to ConnectToRunningBrowser

One more, still in suite_Smoke/desktopui_WindowManagerFocusNewWindows:

http://chromegw/i/chromiumos/builders/x86%20generic%20PFQ/builds/1042/steps/VMTest/logs/stdio

FYI: this issue may have closed the tree 3 times today, so it'll be great once the underlying cause is found and fixed :-)

We are still trying to understand what is wrong. Just landed r103057 which would give us more info if we stuck with offline/portal message when loading Gaia. Tomorrow bots should have this change and we could dig more info out of the failures.

This might be a new instance of this problem:
http://chromegw/i/chromiumos/builders/x86%20generic%20PFQ/builds/1056/steps/VMTest/logs/stdio

========== ERROR FILE /tmp/cbuildbotfh9i7g/test_harness/SimpleTestUpdateAndVerify/2_verify/suite_Smoke/login_CryptohomeUnmounted/debug/login_CryptohomeUnmounted.ERROR FOR TEST suite_Smoke/login_CryptohomeUnmounted ==============
09/28 10:01:56 ERROR|  warnings:0029| /usr/local/lib/python2.6/site-packages/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
09/28 10:01:56 ERROR|  warnings:0029|   super(Connection, self).__init__(*args, **kwargs)

@57 is 20887. Test automation tried to login, we see chrome submitted the form but we don't see the POST in local auth server and automation eventually timed out.

Tree closurer w/ similar symptoms to this or 20887

http://build.chromium.org/p/chromiumos/builders/x86%20generic%20PFQ/builds/1065/steps/cbuildbot/logs/stdio

@59 is 20323 as "WaitForInitialLoads" is present in VMTest log.

My r103057 log captures the navigation error:
[3578:3578:122273393:INFO:CONSOLE(3193)] "Gaia frame error = 501", source: chrome://oobe/login (3193)

Error 501 defined here:
// The server's response was insecure (e.g. there was a cert error).
NET_ERROR(INSECURE_RESPONSE, -501)

So looks like our fakeDNS and injected fake cert does not cheat Chrome and it refuses to load the page.

Ok!  I wonder why only some of the time?

another instance with "WaitForInitialLoads failed" in the log:

http://chromegw/i/chromiumos/builders/x86%20generic%20PFQ/builds/1067/steps/VMTest/logs/stdio

I just recalled that Chrome now has a "--gaia-host" command arg and we could use that to replace "www.google.com" to something else.

Would that be helpful to workaround the cert problem?

It's possible.  Does your code honor that?  It might be a good workaround for now to get rid of this flake and allow me to work on this.

WebUI login code should respect that. The flag is added when need to test against gaiastaging.

Several other instances with:
    Unhandled JSONInterfaceError: Automation call {..., 'command': 'Login'} received empty response.

http://build.chromium.org/p/chromiumos/builders/x86%20generic%20full/builds/477/steps/VMTest/logs/stdio
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-mario%20canary/builds/1067/steps/VMTest/logs/stdio
http://build.chromium.org/p/chromiumos/builders/x86%20pineview%20full/builds/472/steps/VMTest/logs/stdio

@65 Id still have to do some cert stuff to make that work; the subject of the fake cert is 'www.google.com', so the networking stack rejects going to https://127.0.0.1/accounts/ServiceLogin and seeing that cert.

I may use this to loop the DNS stuff out temporarily, if it seems like that'd help.

One more "WaitForInitialLoads failed". It seems randomly happens, not always.

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/stumpy-binary/builds/873/steps/VMTest/logs/stdio


========== ERROR FILE /tmp/cbuildbotgBSCnY/test_harness/testUpdateKeepStateful/2_verify/suite_Smoke/desktopui_WindowManagerFocusNewWindows/debug/desktopui_WindowManagerFocusNewWindows.ERROR FOR TEST suite_Smoke/desktopui_WindowManagerFocusNewWindows ==============
10/02 21:23:53 ERROR|  warnings:0029| /usr/local/lib/python2.6/site-packages/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
10/02 21:23:53 ERROR|  warnings:0029|   super(Connection, self).__init__(*args, **kwargs)
10/02 21:24:41 ERROR|logging_ma:0560| [3775:3775:1002/212441:146640934:ERROR:proxy_launcher.cc(109)] WaitForInitialLoads failed.
10/02 21:24:41 ERROR|logging_ma:0560| [3775:3775:1002/212441:146641048:ERROR:proxy_launcher.cc(562)] Failed to ConnectToRunningBrowser

@68 is still the cert issue.

[4011:4011:102288903:INFO:CONSOLE(3274)] "Gaia frame error = 501", source: chrome://oobe/login (3274)

And again:

========== ERROR FILE /tmp/cbuildbotGm9UZO/test_harness/testUpdateWipeStateful/2_verify/suite_Smoke/desktopui_ScreenLocker/debug/desktopui_ScreenLocker.ERROR FOR TEST suite_Smoke/desktopui_ScreenLocker ==============
10/03 12:57:42 ERROR|  warnings:0029| /usr/local/lib/python2.6/site-packages/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
10/03 12:57:42 ERROR|  warnings:0029|   super(Connection, self).__init__(*args, **kwargs)
10/03 12:58:30 ERROR|logging_ma:0560| [3281:3281:1003/125830:128454122:ERROR:proxy_launcher.cc(109)] WaitForInitialLoads failed.
10/03 12:58:30 ERROR|logging_ma:0560| [3281:3281:1003/125830:128454232:ERROR:proxy_launcher.cc(562)] Failed to ConnectToRunningBrowser

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex-binary/builds/1560/steps/VMTest/logs/stdio

Sheriffs may want to refer to  issue 20887  as well.

Happened again on stumpy canary:

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/stumpy%20canary/builds/403/steps/VMTest/logs/stdio

========== ERROR FILE /tmp/cbuildbotP9lXUG/test_harness/testUpdateWipeStateful/4_verify/suite_Smoke/desktopui_WindowManagerFocusNewWindows/debug/desktopui_WindowManagerFocusNewWindows.ERROR FOR TEST suite_Smoke/desktopui_WindowManagerFocusNewWindows ==============
10/03 13:01:42 ERROR|  warnings:0029| /usr/local/lib/python2.6/site-packages/dbus/connection.py:242: DeprecationWarning: object.__init__() takes no parameters
10/03 13:01:42 ERROR|  warnings:0029|   super(Connection, self).__init__(*args, **kwargs)
10/03 13:02:30 ERROR|logging_ma:0560| [3768:3768:1003/130230:129898656:ERROR:proxy_launcher.cc(109)] WaitForInitialLoads failed.
10/03 13:02:30 ERROR|logging_ma:0560| [3768:3768:1003/130230:129898775:ERROR:proxy_launcher.cc(562)] Failed to ConnectToRunningBrowser

Commit: c15a7a62097c447ed64ea1ed93db8b5d743668c4
 Email: cmasone@chromium.org

[assets] Add custom domain for gaia auth during test

We're having SSL issues faking out www.google.com, so we're
going to start testing against the made-up domain 'insecure.com'

BUG=chromium-os:20323
TEST=suite_Smoke

Change-Id: I9ea166e2a4f0783df570a45de11613bf0250871a
Reviewed-on: http://gerrit.chromium.org/gerrit/8657
Tested-by: Chris Masone <cmasone@chromium.org>
Reviewed-by: Xiyuan Xia <xiyuan@chromium.org>

M	gaia_auth/test/content.js

Still tree closer:
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-zgb%20canary/builds/883

Cert issue:
[28175:28175:415688780:INFO:CONSOLE(3276)] "Gaia frame error = 501", source: chrome://oobe/login (3276)

Commit: 9953aa3fafcbf323d44da5fc44153698b3769ddf
 Email: cmasone@chromium.org

[crosutils] Make auth-testing cert CN be *.google.com

It seems that the intermittent failure we see on the login screen
is due to a sort of mixed-content error, where there's a failure
within the page to load some embedded resource due to a cert error.

Make the SSL cert we inject work for any google.com server to
work around this.

BUG=chromium-os:20323
TEST=login_CryptohomeMounted 50x, suite_Smoke

Change-Id: Ic8e5b9bec799cd19cccfeddd2990fe3a494d2184
Reviewed-on: http://gerrit.chromium.org/gerrit/8818
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Ready: Chris Masone <cmasone@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>

M	mod_for_test_scripts/710enableAuthTesting

Commit: b86e1acb3a410014294ae613397e256972fcec9c
 Email: jglasgow@chromium.org

flimflam: portal: fix timeout computation, use curl_multi_perform

Fix the timeout computation when tv_nsec overflows.  This might cause
incorrect computation of expiration times.

Use curl_multi_perform instead of curl_multi_socket_action with
CURL_SOCKET_TIMEOUT because the former causes curl to handle timed out
requests properly, whereas curl_multi_socket_action does not complete
the timedout requests.

BUG=chromium-os:20323,chromium-os:21127
TEST=while ssh root@jglinux1.cam -p 9222 -o StrictHostKeyChecking=no /usr/local/lib/flimflam/test/list-services | grep online ; do sudo pkill kvm ; ./bin/cros_start_vm  ; sleep 15; done

Reviewed-on: http://gerrit.chromium.org/gerrit/8137
Commit-Ready: Jason Glasgow <jglasgow@chromium.org>
Reviewed-by: Jason Glasgow <jglasgow@chromium.org>
Tested-by: Jason Glasgow <jglasgow@chromium.org>
(cherry picked from commit 70ebe9d77ad247e1d370c3f21f20b1ea96832e6b)

Change-Id: I0489b7bbc250b9b81dcca69b39d0c65de8f8bcdd
Reviewed-on: http://gerrit.chromium.org/gerrit/8864
Reviewed-by: Vincent Palatin <vpalatin@chromium.org>
Tested-by: Jason Glasgow <jglasgow@chromium.org>

M	src/portal.c

I think this and 20887 are dups, but not confident enough to merge them just yet.

This seems to be caused by a DNS request escaping our iptables rule and going out to the real-live internet, causing the test harness to be talking to our Gaia mock the vast majority of the time, but real Gaia for one HTTPS request.  This breaks things, obviously.

another instance of this:

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex-binary/builds/1585/steps/VMTest/logs/stdio

not sure about the "Couldn't find user 'tcpdump'"


========== ERROR FILE /tmp/cbuildbotbqEKA6/test_harness/testUpdateWipeStateful/2_verify/suite_Smoke/desktopui_WindowManagerFocusNewWindows/debug/desktopui_WindowManagerFocusNewWindows.ERROR FOR TEST suite_Smoke/desktopui_WindowManagerFocusNewWindows ==============
10/07 15:17:39 ERROR|base_utils:0113| [stderr] tcpdump: Couldn't find user 'tcpdump'
10/07 15:18:26 ERROR|logging_ma:0560| [4193:4193:1007/151826:140485620:ERROR:proxy_launcher.cc(109)] WaitForInitialLoads failed.
10/07 15:18:26 ERROR|logging_ma:0560| [4193:4193:1007/151826:140485696:ERROR:proxy_launcher.cc(562)] Failed to ConnectToRunningBrowser

Commit: 2f932d2893a3b8eef1d35818a33cbdc737255f8d
 Email: cmasone@chromium.org

[autotest] Switch to resolv.conf munging in lieu of iptables-fu

IPTables is letting us down.  Every now and again, a DNS resolution request
gets through, leading to a botched webui login flow, leading to test failure.

Punt on iptables and move to directly munging /var/run/flimflam/resolv.conf
(which we find by following the symlink at /etc/resolv.conf)

BUG=chromium-os:20323, chromium-os:20887
TEST=suite_Smoke, CryptohomeMounted x 50 on hardware and VM

Change-Id: Ibd0725fc2176dc7cedc46994f8299b0b1b33536e
Reviewed-on: http://gerrit.chromium.org/gerrit/9813
Commit-Ready: Chris Masone <cmasone@chromium.org>
Reviewed-by: Chris Masone <cmasone@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>

M	client/cros/cros_ui_test.py

Will keep an eye on this

Nope.  A real segfault is a real segfault and should be tracked in the issue related to the crash.  None of the reports in this issue had a segv associated with them.

You're right cmasone, I removed my comment.  Will open another bug, even though it's a supplied_chrome bug i.e. no logs. :/

Commit: 44b0bf7c0b8f1c8dc82e08b59c77bbc09e467cfa
 Email: sleffler@chromium.org

chromiumos-overlay: baselayout: add tcpdump user to passwd

The tcpdump package needs a "tcpdump" user to drop privileges.  Since this
package is included only for dev builds it cannot alter the password
file so we reserve/add the user manually here.

BUG=chromium-os:20323, 20572
TEST=build an image and check the contents of /etc/passwd; run tcpdump as root and note it doesn't complain "Couldn't find user tcpdump"

Change-Id: Iaf5fa4ee34df7f77edb327afaff1e70c58ba4bf2
Reviewed-on: http://gerrit.chromium.org/gerrit/9902
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Ready: Sam Leffler <sleffler@chromium.org>
Tested-by: Sam Leffler <sleffler@chromium.org>

D	sys-apps/baselayout/baselayout-2.0.1-r223.ebuild
A	sys-apps/baselayout/baselayout-2.0.1-r224.ebuild

Declaring victory after several days of no repro!

woot!
Many lives were lost of this. Thank you!

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)