(No comment was entered for this change.)

All this means is that a random SEGV has snuck into chrome :-/

I don't know what we're going to be able to do about it

this shouldn't be a tree closure?

I dunno...I mean, it is telling us that Chrome crashes while we ran this test (which logs in and ensures that Chrome can bring up an empty tab).  That's a real, but probably transient, failure.  Should that close our tree?  Thats kinda a policy decision...

Let's see if the next binary builder passes.

Binary builder passed. it is a flaky test.

(No comment was entered for this change.)

(No comment was entered for this change.)

Seeing this on TOT PFQ too.
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/TOT%20Pre-Flight%20Queue/builds/3869/steps/Test/logs/stdio

(No comment was entered for this change.)

whoa...what?  How did this wind up assigned to me?  Did someone roll chrome back to a previous version?  Did doing so indicate which version introduced the issue?

This particular random crash seems to have cleared.  Going to spend some time today working on a tool to help sheriffs get the minimum data that someone would need to investigate a problem such as this, and document the process of gathering that info and assigning an issue to the right person.

(No comment was entered for this change.)

I think it just happened again?

http://chromegw.corp.google.com/i/chromiumos/builders/x86%20generic%20pre%20flight%20queue/builds/6516/steps/Test/logs/stdio

Well, SOME chrome crash happened again :-)

I don't think it makes sense to have this issue open when it is totally unactionable.  Unless someone is planning to update this issue with version information indicating which version of chrome introduced this crash --  and what the last-known-good-version was, no one is going to take any action on this bug.

Ported over from 13758:
http://chromeos-botmaster.mtv.corp.google.com:8026/builders/TOT%20Pre-Flight%20Queue/builds/3923/steps/Test/logs/stdio

Crashes detected during testing:
----------------------------------------------------
supplied_chrome sig 11
  suite_Smoke/login_CryptohomeMounted
----------------------------------------------------

2011-07-22T07:16:02.169147+00:00 localhost logger: cros/login.py: Attempting logout...
2011-07-22T07:16:02.624044+00:00 localhost kernel: [  141.623915] chrome[9849]: segfault at 0 ip   (null) sp 7fa771fc error 14 in .org.chromium.Chromium.IagA9S (deleted)[67fc0000+20000]


So it happens during logout.  Not sure what that last bit means.

It happened once more today:

http://buildbot.jail.google.com/i/chromiumos/builders/x86%20generic%20pre%20flight%20queue/builds/6542/steps/Test/logs/stdio

Snippet:

Crashes detected during testing:
----------------------------------------------------
supplied_chrome sig 11
  suite_Smoke/desktopui_ChromeFirstRender
----------------------------------------------------
Total unique crashes: 1

And again:

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-zgb-binary/builds/497/steps/Test/logs/stdio

Crashes detected during testing:
----------------------------------------------------
supplied_chrome sig 11
  suite_Smoke/login_LoginSuccess.default
----------------------------------------------------
Total unique crashes: 1

Another one:

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/TOT%20Pre-Flight%20Queue/builds/3948/steps/Test/logs/stdio

 Issue 18158  has been merged into this issue.

Another instance:

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/TOT%20Pre-Flight%20Queue/builds/4069/steps/Test/logs/stdio

Similar to the case in comment 15, this happened at logout. From test_harness/SimpleTestUpdateAndVerify/2_verify/suite_Smoke/login_CryptohomeUnmounted/sysinfo/var/log/messages in the build artifacts:

2011-07-27T11:53:00.185698-07:00 localhost logger: cros/login.py: Attempting logout...
2011-07-27T11:53:00.470095-07:00 localhost entdwife.sh: Username: <omitted>
2011-07-27T11:53:00.508255-07:00 localhost entdwife.sh: Extension path is not a directory: /home/chronos/user/Extensions
2011-07-27T11:53:01.028063-07:00 localhost kernel: [  168.027517] chrome[11157]: segfault at 1 ip 00000001 sp 7fb0b2cc error 14

Seen on x86-generic-pfq:

http://build.chromium.org/p/chromiumos/builders/x86%20generic%20pre%20flight%20queue/builds/6662/steps/cbuildbot/logs/stdio

chrome sig 11
  suite_Smoke/login_CryptohomeMounted
supplied_chrome sig 11
  suite_Smoke/desktopui_ChromeFirstRender

Re: comment #14

I think this could be actionable if we were more knowledgeable of what goes on in the chrome side with all these tests.

I tried several times to submit this patch:
http://git.chromium.org/gitweb/?p=chromiumos/platform/cros.git;a=commit;h=1d59bcc7e30e38adfc1293c4cfa79a7c57fccb74

The problem ended up being this bit of code that I had to fix:

 extern "C"
 void ChromeOSDisconnectPowerStatus(PowerStatusConnection connection) {
-  if (connection) {
+  if (connection && connection->connection()) {
     dbus::Disconnect(connection->connection());
     delete connection;
   }

It was giving the chrome sig 11 fault on desktopui_ChromeFirstRender as well as other tests, but consistently with the patch, and never without the patch.

Although I found the location of the bug and fixed it, I can't explain why it didn't happen before I applied my patch, since the data used in that piece of code didn't change.  Based on that, I suspect there are other such "ghost bugs" in libcros that don't manifest themselves consistently, and that's what's causing these problems.

(No comment was entered for this change.)

11:35:54 INFO | 		INFO	login_CryptohomeMounted	Received crash notification for chrome[4779] sig 11 (developer build - not testing - always dumping)	timestamp=1312050954	localtime=Jul 30 11:35:54	
11:35:54 INFO | 		GOOD	login_CryptohomeMounted	login_CryptohomeMounted	timestamp=1312050954	localtime=Jul 30 11:35:54	completed successfully
11:35:55 INFO | 	END GOOD	login_CryptohomeMounted	login_CryptohomeMounted	timestamp=1312050954	localtime=Jul 30 11:35:54	


and again on the pfq:

12:02:25 INFO | Bundling /build/x86-generic/usr/local/autotest/client/site_tests/desktopui_ChromeFirstRender into test-desktopui_ChromeFirstRender.tar.bz2
12:02:40 INFO | 		INFO	desktopui_ChromeFirstRender	Received crash notification for supplied_chrome[2336] sig 11 (developer build - not testing - always dumping)	timestamp=1312052561	localtime=Jul 30 12:02:41	
12:02:41 INFO | 		GOOD	desktopui_ChromeFirstRender	desktopui_ChromeFirstRender	timestamp=1312052561	localtime=Jul 30 12:02:41	completed successfully
12:02:41 INFO | 	END GOOD	desktopui_ChromeFirstRender	desktopui_ChromeFirstRender	timestamp=1312052561	localtime=Jul 30 12:02:41	

I am trying to find more information about the crash, but the test results do not appear to be getting bundled correctly, or are not accessable:

When accessing: http://chromeosbuild7.mtv.corp.google.com/archive/x86-mario-release/0.15.832.0-a1-b803/test_results.tgz

"You don't have permission to access /archive/x86-mario-release/0.15.832.0-a1-b803/test_results.tgz on this server."

I took a look at these test result:
http://chromeosbuild5.mtv.corp.google.com/archive/x86-zgb-private-bin/0.15.837.0-a1-b642/test_results.tgz

I did not see any crash dump in test_harness/testUpdateWipeStateful/4_verify/suite_Smoke/desktopui_ChromeFirstRender

Might it be located elsewhere? What is the name of the file?

cmasone: if you point me to the script that is supposed to copy the crash dump file, you can assign this back to me and I will try to debug it.

 Issue 17898  tracks getting the crash dumps, which thieu and mkrebs are looking at, as thieu wrote the stack generation code and mkrebs wrote the minidump-generating code.  All I did was write some python to create a flag file and copy a directory or two at logout in the interests of trying to help out.

Please work with them and post further updates on the issue that tracks the work in progress -- crosbug.com/17898

(No comment was entered for this change.)

Here are some logs from real hardware:

http://goto/chromeos-dash/emails/test/x86-alex-r15/iAekI.html
http://cautotest:15000/results/51885-chromeos-test/172.31.26.111/login_CryptohomeMounted/sysinfo/messages

2011-08-02T22:11:19.189067-07:00 localhost kernel: [  188.188194] chrome[6129]: segfault at 0 ip   (null) sp 7fc2336c error 14
2011-08-02T22:11:19.280164-07:00 localhost crash_reporter[6480]: Received crash notification for chrome[6129] sig 11 (developer build - not testing - always dumping)

The problem with collecting crash dumps is they are generated during logout and the tests are usually logged in via the guest account. The core dumps are lost when the cryptohome unmounts and is destroyed.

The best way to reproduce this issue with core dumps is to modify a test to use a real login and then run the test repeatedly. When a crash occurs you can then login and manually remount the cryptohome.

I looked a number of example kernel segfault messages from failed tests and usually they are useless indicating running code at zero like this one (probably calling a null function pointer) - I was hoping we could get an offset in chrome to at least know what function that crashed was.  I think mkrebs is adding code so that all crashes during tests are routed to /var/spool/crash.

Again in x86-zgb-canary

Crashes detected during testing:
----------------------------------------------------
chrome sig 11
  suite_Smoke/login_CryptohomeMounted
----------------------------------------------------
Total unique crashes: 1

Also in x86-pineview-full

Crashes detected during testing:
----------------------------------------------------
chrome sig 11
  suite_Smoke/login_CryptohomeMounted
----------------------------------------------------
Total unique crashes: 1

Commit: b09c335f4837e45b3496728703493d68ae7eca1c
 Email: kmixter@chromium.org

chromeos-login: Increase wait for Chrome to shutdown to 1m during tests

BUG=chromium-os:17899
TEST=kill -STOP <chrome-browser-pid>
pkill session_manager
look in /var/log/messages for message about killing process

I see no messages though.

Change-Id: I1070299151be52892fb259e28b6bf32328104fef
Reviewed-on: http://gerrit.chromium.org/gerrit/5289
Reviewed-by: Chris Masone <cmasone@chromium.org>
Tested-by: Ken Mixter <kmixter@chromium.org>

M	session_manager_service.cc

(No need to Restrict-View-Commit here.)

Again in x86-mario canary
(http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-mario%20canary/builds/837/steps/Test/logs/stdio)

Crashes detected during testing:
----------------------------------------------------
chrome sig 11
  suite_Smoke/login_CryptohomeMounted
----------------------------------------------------

Again in TOT PFQ
(http://chromeos-botmaster.mtv.corp.google.com:8026/builders/TOT%20Pre-Flight%20Queue/builds/4395/steps/Test/logs/stdio)

Crashes detected during testing:
----------------------------------------------------
supplied_chrome sig 11
  suite_Smoke/login_CryptohomeMounted
----------------------------------------------------

Again in x86 pineview full
(http://build.chromium.org/p/chromiumos/builders/x86%20pineview%20full/builds/47/steps/Test/logs/stdio)

Crashes detected during testing:
----------------------------------------------------
chrome sig 11
  suite_Smoke/login_CryptohomeMounted
----------------------------------------------------

Again in x86 generic full
(http://build.chromium.org/p/chromiumos/builders/x86%20generic%20full/builds/48/steps/Test/logs/stdio)

Crashes detected during testing:
----------------------------------------------------
chrome sig 11
  suite_Smoke/login_CryptohomeMounted
----------------------------------------------------

even another one in x86 generic pfq
http://chromegw.corp.google.com/i/chromiumos/builders/x86%20generic%20PFQ/builds/139/steps/Test/logs/stdio

chrome sig 11
  suite_Smoke/login_CryptohomeMounted

Here are stacktraces from http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-mario%20canary/builds/837 and http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-mario%20canary/builds/838

FYI, here is the recipe I used for generating stacktrace:
  $ wget http://chromeosbuild7.mtv.corp.google.com/archive/x86-mario-release/0.15.871.0-a1-b838/test_results.tgz
  $ wget http://chromeosbuild7.mtv.corp.google.com/archive/x86-mario-release/0.15.871.0-a1-b838/debug.tgz
  $ tar -xzf test_results.tgz
  $ tar -xzf debug.tgz
  $ ( for i in $(find test_harness -name "chrome*.dmp"); do minidump_stackwalk $i debug/breakpad; done ) > stack.txt 2>/dev/null

stack-b837.txt 35.6 KB   View   Download

stack-b838.txt 42.6 KB   View   Download

Sorry to poke my nose into something I don't understand, but the bug looks like a race between login and prefs setup.

LoginUtilsImpl::OnProfileCreated is called after prefs deserialization of info off disk happens, and it dereferences delegate_.  But delegate_ is only initialized after login succeeds.

This should probably be owned by whoever wrote LoginUtilsImpl.  It seems dangerous to me for it to be possible to create the class without a delegate, the API should disallow that.

LoginUtilsImpl::OnProfileCreated crash is now tracked in  Issue 18269 .

davemoore said that stack-b838 is an example of this issue, so I've attached a few more stack traces of the same problem.

1. stack-alex-he-b17.txt (from http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-alex_he%20canary/builds/17)
2. stack-stumpy-b175.txt (from http://chromeos-botmaster.mtv.corp.google.com:8026/builders/stumpy%20canary/builds/175 )

stack-alex-he-b17.txt 54.2 KB   View   Download

stack-stumpy-b175.txt 42.0 KB   View   Download

OK, this this looking like a race in MessageLoop::DeletePendingTasks() successfully deleting the task out from under the base::subtle::TaskClosureAdapter::Run via inlining of MessageLoop::Post*Task().

There's no lock or obvious atomicity protecting the aliasing for attempting to run the task, and if the race is lost, the pointer could be zero from the element being gone from the list.

Note that a search indicates that this issue goes way back, and transiently gets better or worse depending in the phase of the moon.

Doing a bit of logfile sifting:

zgb-canary r543 is the first reported bad revision, which is also when we moved from chrome 826 (92995) to chrome 827 (93179). All is quiet then until chrome 839 (94841) at zgb-canary r590. That persisted until zgb-canary r595 (at which we moved to chrome 840 =  94889). The next failure is at zgb-canary r623 (chrome 846 = 95736), which persists through now (r631, chrome 847 = 95976).

zgb-canary.11 1.0 KB   View   Download

See also 11407, from last January. Also 13758, 16451 and similar.

I believe that this comes and goes as timing changes the relative order of operations on the task list.

Dave, could you find someone to focus on this?  It is one of the primary reasons for tree closures (the other is 18269, but that's already assigned to one of the MSK team)

Throwing darts, but is issue chromiun:92297 related?

Some snippets to summarize that bug
"This is sort of multiprofile related, since it's not clear that TaskManager, which is a browser process singleton, should be tied to a specific Profile. It's using a renderer attached to a specific Profile in order to render the WebUI, but the backing data is not Profile-specific. I guess it could be argued that the TaskManager should use the Profile of whatever window it belongs to to render the WebUI, which seems reasonable to me. In any case, this is a TaskManager on ChromeOS bug, and perhaps a more general problem for our WebUIs that aren't tied to specific Profiles."
...
"Profile-agnostic WebUIs are outliving the Profile whose renderer they're using, which leads to shutdown crashes."

Okay, so this one is just tracking the stack traces from #43, about crashes in TaskClosureAdapter::Run().

I don't see anything to indicate that  issue 11407  (AudioMixerAlsa singleton hanging on to a stale pointer),  issue 13758  (useless bug just saying that Chrome segfaulted), or issue 16451 (compositing?  sqlite?) is related to this.

#49: Not sure, but the stack traces and description of the problem in http://crbug.com/92297 look pretty different from the segfaults in #43.

I'll try to take a look at this soon but don't mind if anyone else wants to grab it or help out.

(No comment was entered for this change.)

I don't have any ideas from looking at the stack traces or the code.  Is this happening consistently now?  Figuring out which task we're trying to run when the crash occurs might provide some useful clues.  tracked_objects::Location provides a relatively easy way of getting this.

It seems to happen pretty consistently, yes.  Running some test that logs in and logs out (RemoteLogin?) over and over should repro it eventually

I'm currently trying to do the stuff I described in #52.  Our build system is not cooperating.

Commit: b1966e1eecb4edd45af1dad1c3237a38e90e8ca5
 Email: rochberg@chromium.org

Put in framework to Temporarily ignore crashes

Chrome sig11's are to the point where they're obscuring other bugs.
Now that we have someone who owns this bug (derat@), I'd like to turn
off the notifications for a while to make sheriffing a little easier.

Several people have expressed concern that by ignoring these sorts of
things, we'll end up sweeping them under the rug and forgetting them.
Hence, I've added a deadline for this and future exceptions.

This CL also removes two old exceptions (for syntpenh and the window
manager).  These bugs aren't fixed (or they weren't the last time Dale
looked), but they've been languishing in obscurity.

BUG=chromium-os:17899
TEST=Ran by hand against downloaded build artifacts.

Change-Id: Id6e2fc4dbf0026e8a53dfaed5ab187058d049c69
Reviewed-on: http://gerrit.chromium.org/gerrit/5755
Reviewed-by: Chris Masone <cmasone@chromium.org>
Tested-by: David Rochberg <rochberg@chromium.org>

M	utils_py/generate_test_report.py

Adding a stack trace from the crash on x86-pineview-full build 79 that seems to show this signature.

I have been trying to grab a stack trace when a chrome sig11 closes the tree in case we're overlooking other issues.  Let me know if these are not useful and I'll stop adding them to bugs.

stack.txt 55.7 KB   View   Download

Commit: 6db3f80ab78c4cb2d3b09378add10236dd412153
 Email: derat@chromium.org

autotest: Wait for login to finish before logging out.

This makes login.attempt_logout() call
wait_for_initial_chrome_window() before asking the session
manager to stop the session.  I'm hoping that this will
avoid a bunch of Chrome and window manager crashes that
we've seen when the UI job is stopped soon after it's been
started.

BUG=chromium-os:17899,chromium-os:18269
TEST=ran suite_Smoke a few times without seeing any chrome crashes (i was seeing them consistently in login_CryptohomeMounted and login_CryptohomeUnmounted before)

Change-Id: I80d3d6aaa0db5b7a12f973d114270a7f23d324a1
Reviewed-on: http://gerrit.chromium.org/gerrit/5797
Reviewed-by: Dave Moore <davemoore@chromium.org>
Tested-by: Daniel Erat <derat@chromium.org>

M	client/cros/login.py

I've checked in http://gerrit.chromium.org/gerrit/5797 , which seems to prevent Chrome segfaults locally.  We'll see.

The underlying bug in Chrome should still be fixed, but I haven't had much luck tracking it down:
- I don't know of any way to get Chrome to run under a debugger while the tests are running -- they're constantly starting and stopping it.
- I added some print statements to MessageLoop to try to figure out which task it was trying to run when it crashed, but the last message in the log was just a "done running task" message -- perhaps the message for the failing task got buffered.
- Converting between the user-readable-times and time_t values printed by the testing code and the uptime-in-microseconds values from Chrome is painful.

I got distracted with the testing-based workaround described above at this point.

Like comment 56, I'm attaching a stack trace from a crash on x86-pineview-full  
http://chromegw/i/chromiumos/builders/x86%20pineview%20full/builds/82 that seems to show this signature.

b82.txt 51.0 KB   View   Download

Re comment #59, it looks more like  bug 18269 .  I've copied the stack trace over there.

You will have difficulty reproducing this outside a QEMU environment.

Anecdotally, the machines which run the Chrome-OS tests in this area (the login check) do so in a QEMU, and there has been increasing load on these machines, which has in turn led to timing changes.  The timing changes expose the bug.

You have to look at the MessageLoop code in the context of a shutdown as a result of an out-of-thread event (like a signal, another thread, etc.) happening at the same time that the Task is in the process of being Alias'ed + dispatched.  The race seen via source inspection happens there (it takes the class, and uses it outside the pattern).  There's no barrier to pull the element off temporarily instead of aliasing it.

I think that this is lower priority now (assuming that it's no longer showing up after #57).

Will probably knows the most about this code.  Relevant stack traces are the two in comment #43.

(No comment was entered for this change.)

Zel, can I merge http://gerrit.chromium.org/gerrit/5797 to R14?

(No comment was entered for this change.)

Merged to 811 as http://gerrit.chromium.org/gerrit/6022.

Commit: dd1ecd80d2b320ac25cb382342321f84fc098283
 Email: derat@chromium.org

autotest: Wait for login to finish before logging out.

This makes login.attempt_logout() call
wait_for_initial_chrome_window() before asking the session
manager to stop the session.  I'm hoping that this will
avoid a bunch of Chrome and window manager crashes that
we've seen when the UI job is stopped soon after it's been
started.

BUG=chromium-os:17899,chromium-os:18269
TEST=ran suite_Smoke a few times without seeing any chrome crashes (i was seeing them consistently in login_CryptohomeMounted and login_CryptohomeUnmounted before)

Change-Id: I80d3d6aaa0db5b7a12f973d114270a7f23d324a1
Reviewed-on: http://gerrit.chromium.org/gerrit/5797
Reviewed-by: Dave Moore <davemoore@chromium.org>
Tested-by: Daniel Erat <derat@chromium.org>
Reviewed-on: http://gerrit.chromium.org/gerrit/6022
Reviewed-by: Daniel Erat <derat@chromium.org>

M	client/cros/login.py

(No comment was entered for this change.)

No longer a TreeCloser thanks to above fix.   http://gerrit.chromium.org/gerrit/5797 helped

(No comment was entered for this change.)

(No comment was entered for this change.)

This bug has not been touched in a while. Please re-open (in crbug.com) if you can repro in M20+. Thanks a lot for your help!

This bug has not been touched in a while. Please re-open (in crbug.com) if you can repro in M20+. Thanks a lot for your help!

(No comment was entered for this change.)

(No comment was entered for this change.)