(No comment was entered for this change.)

(No comment was entered for this change.)

Perhaps mkrebs?  I know he's working on all manner of crash reporter things these days.  

 Issue 17987  has been merged into this issue.

This is critical to debugging Chrome flakiness on tip of tree. Without logs, it is not possible to debug Chrome crashes

The first step is to create /mnt/stateful_partition/etc/enable_chromium_coredumps

I'm not sure where to create this...there doesn't seem to be a good place in an ebuild to do it, off the top of my head.

Taking this over

I have a change pending (http://gerrit.chromium.org/gerrit/#change,4861) that will allow the crash reporter to collect chrome crashes if a certain file exists.  This is basically what Darin suggested in "http://crosbug.com/17987", where Autotest's UITest class could create/remove that file.  If that's useful for this issue, I expect the change to be submitted soon.

http://gerrit.chromium.org/gerrit/#change,4876

I wrote this up today, which touches a different magic file that -- apparently -- we've supported for quite some time now, and causes chrome to actually dump core.  It also copies those core files back over when the test is done.  we'd need to symbolize the core file, but we've already got code that symbolizes minidumps at the end of an autotest run, so it's gotta be doable :-)

I dunno, I tend to favor the solution that does not require any changes outside of autotest, but am mostly indifferent.

Talked to Ken, and I am wrong :-)

My change would disable the collecting of all non-chrome crash dumps :-P

So, I will update the change I made to create the magic file that Mike's code looks for instead. 

(No comment was entered for this change.)

Commit: eb32f939c38d5bc3dad323d5d9e3778b32001c5b
 Email: cmasone@chromium.org

[autotest] Enable Chrome coredumps, copy back to the device after UITests.

Touch the magic file to enable crash_reporter to handle chrome minidumps.

BUG=chromium-os:17898
TEST=manually trigger crash during a test, see that the core files are copied back

Change-Id: Id2b010fcd335e2efa243aefb2b8896cdc5c927ab
Reviewed-on: http://gerrit.chromium.org/gerrit/4876
Tested-by: Chris Masone <cmasone@chromium.org>
Reviewed-by: Michael Krebs <mkrebs@chromium.org>

M	client/cros/constants.py
M	client/cros/cros_ui_test.py

 Issue 16513  has been merged into this issue.

 Issue 16514  has been merged into this issue.

 Issue 16515  has been merged into this issue.

Commit: 9e8e5f468993a49b0004a3bcef5c81b9e1fe1f63
 Email: mkrebs@chromium.org

Don't ignore chrome crashes when autotesting.

If the file "/mnt/stateful_partition/etc/collect_chrome_crashes" exists on
the target, allow chrome crashes to be reported by crash reporter.  The
intention is that the autotest system will create this file.  This is a
relatively temporary fix to help track down chrome problems, until the
crashes can be handled by crash reporter in the field as well.

BUG=chromium-os:17987, chromium-os:17898
TEST=Ran unit tests.

Change-Id: I68a584a0b861669117d8e97f5687b4c8fc876011
Reviewed-on: http://gerrit.chromium.org/gerrit/4861
Reviewed-by: Ken Mixter <kmixter@chromium.org>
Tested-by: Ken Mixter <kmixter@chromium.org>

M	user_collector.cc
M	user_collector.h
M	user_collector_test.cc

I pushed Mike's change though there's still a need to generate the /mnt/stateful_partition/etc/collect_chrome_crashes during tests.

I landed a change yesterday to do that :-)

So...we should be good!

Whoops.  Chrome crashes get put in the cryptohome while users are logged in.  So we'll need to copy them out.

(No comment was entered for this change.)

Commit: b0e162534f79f16882088699fc07464c77ac835e
 Email: cmasone@chromium.org

[autotest] Copy crash reports out of cryptohome on every logout

We're losing Chrome crash reports that occur while the user is logged in.
Copy them out so that we can process the minidumps.

BUG=chromium-os:17898
TEST=suite_Smoke

Change-Id: I1faaab8c48b4b9677adae2081c70f0394e1e39e3
Reviewed-on: http://gerrit.chromium.org/gerrit/5045
Tested-by: Chris Masone <cmasone@chromium.org>
Reviewed-by: Gaurav Shah <gauravsh@chromium.org>

M	client/cros/cros_ui_test.py
M	client/site_tests/login_CryptohomeMounted/login_CryptohomeMounted.py
M	client/site_tests/power_LoadTest/power_LoadTest.py

Commit: cafdb0487979ecc4e414213a2aeb170ba20f50e4
 Email: cmasone@chromium.org

[autotest] Ensure we grab chrome crashes when we log out

Missed a couple spots where we call logout()

BUG=chromium-os:17898
TEST=manual, introduce a 'pkill -11 ^chrome$' in the middle of some test that logs in, run it, and ensure that we copy back chrome crash dumps

Change-Id: Ie8abe567c0e7b7fd69755b99bbc6f12763304254
Reviewed-on: http://gerrit.chromium.org/gerrit/5070
Reviewed-by: Chris Masone <cmasone@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>

M	client/cros/cros_ui_test.py

(No comment was entered for this change.)

(No comment was entered for this change.)

As thieu pointed out, grabbing crash reports -before- logging out does not mean we'll get the crashes that occur during log out.

That's harder.  Thieule and mkrebs are working on this now.

(No comment was entered for this change.)

(No comment was entered for this change.)

Commit: e8bd1ae8d444b27a68c1c8da28915a44118a09c6
 Email: mkrebs@chromium.org

Fix logging_UserCrash for recent crash_reporter change.

Don't test that a core file is left behind for developer builds.  Commit
http://gerrit.chromium.org/gerrit/#change,4861 changed crash_reporter to
always remove core files if we're in the middle of testing crash reporting
itself.

BUG=chromium-os:17987, chromium-os:17898
TEST=Ran logging_UserCrash remote test

Change-Id: I08e3995f1ba62f1a5b48613b256dc8b031592498
Reviewed-on: http://gerrit.chromium.org/gerrit/5185
Reviewed-by: Ken Mixter <kmixter@chromium.org>
Tested-by: Michael Krebs <mkrebs@chromium.org>

M	client/site_tests/logging_UserCrash/logging_UserCrash.py

https://sandbox.google.com/storage/chromeos-image-archive/x86-pineview-full/0.15.870.0-a1-b47/_index.html

Hm.  Looks like we got _a_ chrome crash dump, though it's also filed in the wrong test case somehow (login_CryptohomeIncognitoUnmounted instead of login_CryptohomeMounted)

it's not symbolized, though the debug symbols seem to also be available at the link above...I'm confuzzled.  But, perhaps this will at least enable us to get someone to start working on  bug17899 ?

Here are stacktraces from http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-mario%20canary/builds/837 and http://chromeos-botmaster.mtv.corp.google.com:8026/builders/x86-mario%20canary/builds/838

FYI, here is recipe for generating stacktrace:
  $ wget http://chromeosbuild7.mtv.corp.google.com/archive/x86-mario-release/0.15.871.0-a1-b838/test_results.tgz
  $ wget http://chromeosbuild7.mtv.corp.google.com/archive/x86-mario-release/0.15.871.0-a1-b838/debug.tgz
  $ tar -xzf test_results.tgz
  $ tar -xzf debug.tgz
  $ ( for i in $(find test_harness -name "chrome*.dmp"); do minidump_stackwalk $i debug/breakpad; done ) > stack.txt 2>/dev/null

stack-b837.txt 35.6 KB   View   Download

stack-b838.txt 42.6 KB   View   Download

well, the b838 track looks like the bug we've been looking for in 17899 -- but I don't know how much it will help :-/

and b837 looks like this: https://code.google.com/p/chromium-os/issues/detail?id=18269

So, we actually collect these crashes now.  As such, I will mark it fixed.  Continuing work is tracked elsewhere.

crosbug.com/18484 tracks symbolizing them correctly (thieule is on it, he knows what needs doing)
The crashes that originally triggered this are tracked in crosbug.com/17899 and crosbug.com/18269
Associating dumps with correct tests tracked in crosbug.com/18943 (we seem to have some kind of one-off error currently)

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)

(No comment was entered for this change.)