Issue 50060 - chromium - ui_test and browser_tests on mac test bots (10.5 &amp; 10.6) seem to leak processes when things go wrong and take out the bot - An open-source project to help move the web forward.

Issue 50060:	ui_test and browser_tests on mac test bots (10.5 &amp; 10.6) seem to leak processes when things go wrong and take out the bot
	4 people starred this issue and may be notified of changes.	Back to list

Status:	Verified
Owner:	phajdan.jr@chromium.org
Closed:	Dec 2010
Cc:	mark@chromium.org, thomasvl@chromium.org, sky@chromium.org, stuartmorgan@chromium.org, asargent@chromium.org, bauerb@chromium.org, phajdan.jr@chromium.org
Type-Bug Pri-2 OS-Mac ext Verifier-Rohitbm bulkmove TaskForce-GreenTree Cr-Platform-Extensions Cr-Tests-Disabled

Restricted

Only users with Commit permission may comment.

Project Member Reported by thomasvl@chromium.org, Jul 23, 2010

Since getting back from vacation, I've already had to bounce 3 bots in ~ 24 hours for this problem.  What appears to be happening is this:

- Tests are failing to property shutdown the Chromium processes they startup, this leak of processes
- This builds up causing general slow downs causing more tests to fail (and more process to get leaked)
- Eventually the bot runs out of processes and even update steps fail

When I've logged in to check on things, the left over processes can have windows, I see the download confirmation window up (quitting with active downloads), I see the file save dialog asking to confirm saving the something like "download.lib".

I'm gonna start digging through logs to see what causes it, but this is the initial state of info so other folks can find this.

Jul 23, 2010

#1 thomasvl@chromium.org

(No comment was entered for this change.)

Summary: ui_test and browser_tests on mac test bots (10.5 & 10.6) seem to leak processes when things go wrong and take out the bot

Jul 23, 2010

#2 thomasvl@chromium.org

First thing to go south since one bounce: http://build.chromium.org/buildbot/waterfall/builders/Mac10.6%20Tests/builds/5140/steps/browser_tests/logs/stdio, the build summary shows this step as orange but the FAILY_ and FAILS_ tests, but it also says some didn't complete.  The next step http://build.chromium.org/buildbot/waterfall/builders/Mac10.6%20Tests/builds/5140/steps/ui_tests/logs/stdio has the first real failure, after some failing FLAKY_ tests we get:

[ RUN      ] DownloadTest.NewWindow
[23948:263:92848594298757:ERROR:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/first_run_mac.mm(63)] Not implemented reached in static bool FirstRun::ProcessMasterPreferences(const FilePath&, FirstRun::MasterPrefs*)
/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/download/download_uitest.cc:511: Failure
Value of: automation()->WaitForWindowCountToBecome(1)
  Actual: false
Expected: true
[  FAILED  ] DownloadTest.NewWindow (25851 ms)

followed later in that run by:

[----------] 1 test from MultipartResponseUITest
[ RUN      ] MultipartResponseUITest.SingleVisit
[24013:263:92889097948344:ERROR:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/first_run_mac.mm(63)] Not implemented reached in static bool FirstRun::ProcessMasterPreferences(const FilePath&, FirstRun::MasterPrefs*)
[23713:2307:0723/102427:92889307599576:INFO:/b/slave/chromium-rel-mac-builder/build/src/net/test/test_server.cc(208)] Trying to launch python ...
/b/slave/chromium-rel-mac6-tests/build/src/third_party/tlslite/tlslite/utils/cryptomath.py:9: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  import sha
/b/slave/chromium-rel-mac6-tests/build/src/third_party/tlslite/tlslite/mathtls.py:7: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
/b/pylibs/zope/__init__.py:19: UserWarning: Module twisted was already imported from /b/pylibs/twisted/__init__.pyc, but /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python is being added to sys.path
  import pkg_resources
/b/pylibs/zope/__init__.py:19: UserWarning: Module zope was already imported from /b/pylibs/zope/__init__.pyc, but /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python is being added to sys.path
  import pkg_resources
[23713:2307:0723/102428:92890317914672:INFO:/b/slave/chromium-rel-mac-builder/build/src/net/test/test_server.cc(224)] Started on port 1337
[23713:2307:0723/102429:92891858422833:INFO:/b/slave/chromium-rel-mac-builder/build/src/net/test/test_server.cc(272)] Stopped.
../chrome/test/ui/ui_test.h:527: Failure
Value of: server_.get()
  Actual: false
Expected: true
[23713:2307:0723/102429:92891858759545:ERROR:/b/slave/chromium-rel-mac-builder/build/src/base/process_util_posix.cc(105)] Received signal 10
	0   ui_tests                            0x001c5be1 0x0 + 1858529
	1   ui_tests                            0x001e0903 0x0 + 1968387
	2   libSystem.B.dylib                   0x9027e42b _sigtramp + 43
	3   ???                                 0xffffffff 0x0 + 4294967295
	4   ui_tests                            0x001b350c 0x0 + 1783052
	5   ui_tests                            0x001b5100 0x0 + 1790208
	6   ui_tests                            0x001b51cd 0x0 + 1790413
	7   ui_tests                            0x001b5942 0x0 + 1792322
	8   ui_tests                            0x003346e6 0x0 + 3360486
	9   ui_tests                            0x00334b88 0x0 + 3361672
	10  ui_tests                            0x00334d04 0x0 + 3362052
	11  ui_tests                            0x003353d2 0x0 + 3363794
	12  ui_tests                            0x0019f1b7 0x0 + 1700279
	13  ui_tests                            0x0019ceb4 0x0 + 1691316
	14  ui_tests                            0x00002496 0x0 + 9366

the very next run of browser tests shows things failing to get closed (http://build.chromium.org/buildbot/waterfall/builders/Mac10.6%20Tests/builds/5141/steps/browser_tests/logs/stdio):

[ RUN      ] BrowserInitTest.FLAKY_BlockBadURLs
[24527:263:0723/104116:93898014576778:ERROR:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/first_run_mac.mm(63)] Not implemented reached in static bool FirstRun::ProcessMasterPreferences(const FilePath&, FirstRun::MasterPrefs*)
[24527:22787:0723/104116:93898018586551:INFO:/b/slave/chromium-rel-mac-builder/build/src/net/base/host_resolver_impl.cc(1071)] IPv6Probe forced AddressFamily setting to ADDRESS_FAMILY_IPV4
[24527:263:0723/104116:93898021133907:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extensions_service.cc(628)] Sending EXTENSION_LOADED
[24527:263:0723/104116:93898040907735:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/sync/profile_sync_service.cc(103)] Unofficial build, using dev channel sync server.
[24527:263:0723/104116:93898040993556:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/sync/profile_sync_service.cc(129)] Starting ProfileSyncService.
[24527:263:0723/104116:93898041014691:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/sync/profile_sync_service.cc(197)] Using https://clients4.google.com/chrome-sync/dev for sync server URL.
[24527:263:0723/104116:93898041111501:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/sync/profile_sync_service.cc(352)] Clearing Sync DB.
[24527:22019:0723/104116:93898041838948:WARNING:/b/slave/chromium-rel-mac-builder/build/src/net/base/file_stream_posix.cc(350)] Failed to open file: 2 (/private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.EQedmF/test_user_data/Default/Last Tabs)
[24527:263:0723/104116:93898117913513:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/history/history.cc(747)] History backend finished loading
[24527:22019:0723/104116:93898124597473:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/common/important_file_writer.cc(71)] successfully saved /private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.EQedmF/test_user_data/Default/Bookmarks
/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/browser_init_browsertest.cc:86: Failure
Value of: 3
Expected: browser()->tab_count()
Which is: 1
[24527:22787:0723/104116:93898455230427:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/renderer_host/buffered_resource_handler.cc(208)] To buffer: http://linkhelp.clients.google.com/tbproxy/lh/fixurl?hl=en-US&sd=com&url=http%3A%2F%2Fwww.google.com%2F&sourceid=chrome&error=dnserror
[24527:22787:0723/104116:93898457128626:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/renderer_host/buffered_resource_handler.cc(140)] Finished buffering http://linkhelp.clients.google.com/tbproxy/lh/fixurl?hl=en-US&sd=com&url=http%3A%2F%2Fwww.google.com%2F&sourceid=chrome&error=dnserror
[24527:22019:0723/104116:93898804519848:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/common/important_file_writer.cc(71)] successfully saved /private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.EQedmF/test_user_data/Local State
[24527:22019:0723/104116:93898806861877:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/common/important_file_writer.cc(71)] successfully saved /private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.EQedmF/test_user_data/Default/Preferences
[  FAILED  ] BrowserInitTest.FLAKY_BlockBadURLs (830 ms)

Jul 23, 2010

#3 thomasvl@chromium.org

Going back to yesterday in that bots logs (it was bounced for this yesterday also), I see a slow climb in FAILS_ / FLAKY_ test failures before it started having real tests fail, so it could be some of those already bad tests are what's causing the state to collect, but at first blush nothing is jumping out at me as a common pattern between the two for the slow climb.

Jul 23, 2010

#4 thomasvl@chromium.org

Actually, the first errors each time see to start with something with browser tests and failing to load the nibs for extension notifications and then the zombie code catches stale/bad point usage.

Note: Google Test filter = ExtensionApiTest.FLAKY_Notifications
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from ExtensionApiTest
[ RUN      ] ExtensionApiTest.FLAKY_Notifications
[22924:263:0723/101602:92384497115927:ERROR:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/first_run_mac.mm(63)] Not implemented reached in static bool FirstRun::ProcessMasterPreferences(const FilePath&, FirstRun::MasterPrefs*)
[22924:22787:0723/101602:92384500709624:INFO:/b/slave/chromium-rel-mac-builder/build/src/net/base/host_resolver_impl.cc(1071)] IPv6Probe forced AddressFamily setting to ADDRESS_FAMILY_IPV4
[22924:263:0723/101602:92384503585261:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extensions_service.cc(628)] Sending EXTENSION_LOADED
[22924:263:0723/101602:92384524474648:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/sync/profile_sync_service.cc(103)] Unofficial build, using dev channel sync server.
[22924:263:0723/101602:92384524563596:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/sync/profile_sync_service.cc(129)] Starting ProfileSyncService.
[22924:263:0723/101602:92384524585037:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/sync/profile_sync_service.cc(197)] Using https://clients4.google.com/chrome-sync/dev for sync server URL.
[22924:263:0723/101602:92384524690467:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/sync/profile_sync_service.cc(352)] Clearing Sync DB.
[22924:22019:0723/101602:92384526292793:WARNING:/b/slave/chromium-rel-mac-builder/build/src/net/base/file_stream_posix.cc(350)] Failed to open file: 2 (/private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.aSjhpz/test_user_data/Default/Last Tabs)
[22924:263:0723/101602:92384603064628:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/history/history.cc(747)] History backend finished loading
[22924:22019:0723/101602:92384611889354:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/common/important_file_writer.cc(71)] successfully saved /private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.aSjhpz/test_user_data/Default/Bookmarks
[22924:263:0723/101602:92384767979154:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_apitest.cc(67)] Running ExtensionApiTest with: notifications/has_not_permission
[22924:22019:0723/101602:92384768672180:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extensions_service.cc(1220)] Loading single extension from has_not_permission
[22924:263:0723/101602:92384779585845:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extensions_service.cc(628)] Sending EXTENSION_LOADED
[22924:263:0723/101602:92384779723276:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(192)] Creating RenderView for notifications test
[22924:22019:0723/101602:92384779745037:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/common/important_file_writer.cc(71)] successfully saved /private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.aSjhpz/test_user_data/Default/Preferences
Got EXTENSION_LOADED notification.
[22924:263:0723/101602:92384780495282:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(709)] (RenderViewCreated) Resetting EFD to chrome-extension://kcknkhcnfemmaimpkpdlpoidmfdmbdno/background.html for notifications test
[22924:263:0723/101602:92384780556382:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(212)] Request to NavigateToURL chrome-extension://kcknkhcnfemmaimpkpdlpoidmfdmbdno/background.html for notifications test
[22924:263:0723/101602:92384780577936:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(233)] Navigating to chrome-extension://kcknkhcnfemmaimpkpdlpoidmfdmbdno/background.html
[22924:22019:0723/101602:92384780927259:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/common/important_file_writer.cc(71)] successfully saved /private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.aSjhpz/test_user_data/Default/Preferences
[22924:263:0723/101602:92384784040097:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(252)] Sending EXTENSION_PROCESS_CREATED
[22924:263:0723/101602:92384908924520:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(328)] (DidNavigate) Resetting EFD to chrome-extension://kcknkhcnfemmaimpkpdlpoidmfdmbdno/background.html for notifications test
( RUN      ) hasPermission
[22924:263:0723/101603:92384959662047:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_test_api.cc(35)] ( RUN      ) hasPermission
[22924:263:0723/101603:92384960547785:INFO:CONSOLE(0)] "[SUCCESS] hasPermission," source:  (93)
(  SUCCESS )
[22924:263:0723/101603:92384960726963:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_test_api.cc(35)] (  SUCCESS )
[22924:263:0723/101603:92384961565854:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(395)] Sending EXTENSION_HOST_DID_STOP_LOADING
Got EXTENSION_HOST_DID_STOP_LOADING notification.
[22924:263:0723/101603:92384961803246:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_browsertest.cc(243)] All ExtensionHosts loaded
( RUN      ) showNotification
[22924:263:0723/101603:92384962258013:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_test_api.cc(35)] ( RUN      ) showNotification
[22924:263:0723/101603:92384963029917:INFO:CONSOLE(0)] "[SUCCESS] showNotification," source:  (93)
(  SUCCESS )
[22924:263:0723/101603:92384963388937:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_test_api.cc(35)] (  SUCCESS )
[22924:263:0723/101603:92384964478220:INFO:CONSOLE(0)] "All tests succeeded," source:  (53)
Got EXTENSION_TEST_PASSED notification.
[22924:263:0723/101603:92384964799955:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_apitest.cc(67)] Running ExtensionApiTest with: notifications/has_permission_manifest
[22924:22019:0723/101603:92384965113070:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extensions_service.cc(1220)] Loading single extension from has_permission_manifest
[22924:263:0723/101603:92384966987733:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extensions_service.cc(628)] Sending EXTENSION_LOADED
[22924:263:0723/101603:92384967117242:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(192)] Creating RenderView for notifications test
Got EXTENSION_LOADED notification.
[22924:22019:0723/101603:92384967473571:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/common/important_file_writer.cc(71)] successfully saved /private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.aSjhpz/test_user_data/Default/Preferences
[22924:22019:0723/101603:92384969672557:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/common/important_file_writer.cc(71)] successfully saved /private/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.aSjhpz/test_user_data/Default/Preferences
[22924:263:0723/101603:92384969865514:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(709)] (RenderViewCreated) Resetting EFD to chrome-extension://okndaaojedchelofdiaaaejfhofpebon/background.html for notifications test
[22924:263:0723/101603:92384969926690:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(212)] Request to NavigateToURL chrome-extension://okndaaojedchelofdiaaaejfhofpebon/background.html for notifications test
[22924:263:0723/101603:92384969947875:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(233)] Navigating to chrome-extension://okndaaojedchelofdiaaaejfhofpebon/background.html
[22924:263:0723/101603:92384970033499:INFO:CONSOLE(0)] "Uncaught completed," source: chrome/extensionapitest (19)
[22924:263:0723/101603:92384974772451:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(252)] Sending EXTENSION_PROCESS_CREATED
[22924:263:0723/101603:92385097084629:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(328)] (DidNavigate) Resetting EFD to chrome-extension://okndaaojedchelofdiaaaejfhofpebon/background.html for notifications test
( RUN      ) hasPermission
[22924:263:0723/101603:92385147775253:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_test_api.cc(35)] ( RUN      ) hasPermission
[22924:263:0723/101603:92385148617178:INFO:CONSOLE(0)] "[SUCCESS] hasPermission," source:  (93)
(  SUCCESS )
[22924:263:0723/101603:92385148792222:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_test_api.cc(35)] (  SUCCESS )
[22924:263:0723/101603:92385149528789:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_host.cc(395)] Sending EXTENSION_HOST_DID_STOP_LOADING
Got EXTENSION_HOST_DID_STOP_LOADING notification.
[22924:263:0723/101603:92385149758749:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_browsertest.cc(243)] All ExtensionHosts loaded
( RUN      ) absoluteURL
[22924:263:0723/101603:92385150552847:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/extensions/extension_test_api.cc(35)] ( RUN      ) absoluteURL
[22924:263:0723/101603:92385151369046:INFO:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/notifications/notification_ui_manager.cc(62)] Added notification. URL: chrome-extension://okndaaojedchelofdiaaaejfhofpebon/notification.html
2010-07-23 10:16:03.242 browser_tests[22924:107] -[BalloonController loadWindow]: failed to load window nib file 'Notification'.
2010-07-23 10:16:03.243 browser_tests[22924:107] -[BalloonController loadWindow]: failed to load window nib file 'Notification'.
2010-07-23 10:16:03.244 browser_tests[22924:107] -[BalloonController loadWindow]: failed to load window nib file 'Notification'.
[22924:263:0723/101603:92385155792904:ERROR:/b/slave/chromium-rel-mac-builder/build/src/chrome/browser/cocoa/objc_zombie.mm(205)] Zombie <RenderWidgetHostViewCocoa: 0x3190c40> received -frame
[22924:263:0723/101603:92385155860691:ERROR:/b/slave/chromium-rel-mac-builder/build/src/base/process_util_posix.cc(105)] Received signal 10
	0   browser_tests                       0x010c1c81 0x0 + 17570945
	1   browser_tests                       0x010e3e63 0x0 + 17710691
	2   libSystem.B.dylib                   0x9027e42b _sigtramp + 43
	3   ???                                 0xffffffff 0x0 + 4294967295
	4   browser_tests                       0x00a4d033 0x0 + 10801203
	5   CoreFoundation                      0x91dca436 __NSGetForwardingTarget + 86
	6   CoreFoundation                      0x91e6fc70 __forwarding_prep_1___ + 16
	7   browser_tests                       0x00a4bb8a 0x0 + 10795914
	8   browser_tests                       0x00a4afa8 0x0 + 10792872
	9   browser_tests                       0x00a4b925 0x0 + 10795301
	10  browser_tests                       0x00d102e7 0x0 + 13697767
	11  browser_tests                       0x00d1049b 0x0 + 13698203
	12  browser_tests                       0x00d0e300 0x0 + 13689600
	13  browser_tests                       0x00d9e9a8 0x0 + 14281128
	14  browser_tests                       0x00d7fd92 0x0 + 14155154
	15  browser_tests                       0x01b2c36b 0x0 + 28492651
	16  browser_tests                       0x010d872c 0x0 + 17663788
	17  browser_tests                       0x010d893e 0x0 + 17664318
	18  browser_tests                       0x010d8aa7 0x0 + 17664679
	19  browser_tests                       0x01110684 0x0 + 17892996
	20  CoreFoundation                      0x91d8f15b __CFRunLoopDoSources0 + 1563
	21  CoreFoundation                      0x91d8cc1f __CFRunLoopRun + 1071
	22  CoreFoundation                      0x91d8c0f4 CFRunLoopRunSpecific + 452
	23  CoreFoundation                      0x91d8bf21 CFRunLoopRunInMode + 97
	24  HIToolbox                           0x950dd0fc RunCurrentEventLoopInMode + 392
	25  HIToolbox                           0x950dceb1 ReceiveNextEventCommon + 354
	26  HIToolbox                           0x950dcd36 BlockUntilNextEventMatchingListInMode + 81
	27  AppKit                              0x9681a135 _DPSNextEvent + 847
	28  AppKit                              0x96819976 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 156
	29  AppKit                              0x967dbbef -[NSApplication run] + 821
	30  browser_tests                       0x0111002d 0x0 + 17891373
	31  browser_tests                       0x0110f816 0x0 + 17889302
	32  browser_tests                       0x010d9969 0x0 + 17668457
	33  browser_tests                       0x010d9a44 0x0 + 17668676
	34  browser_tests                       0x01095812 0x0 + 17389586
	35  browser_tests                       0x0005dac5 0x0 + 383685
	36  browser_tests                       0x0005e449 0x0 + 386121
	37  browser_tests                       0x000d40c5 0x0 + 868549
	38  browser_tests                       0x0015e373 0x0 + 1434483
	39  browser_tests                       0x00965df8 0x0 + 9854456
	40  browser_tests                       0x0015d27c 0x0 + 1430140
	41  browser_tests                       0x0118174f 0x0 + 18356047
	42  browser_tests                       0x01181c68 0x0 + 18357352
	43  browser_tests                       0x01181de4 0x0 + 18357732
	44  browser_tests                       0x011824b2 0x0 + 18359474
	45  browser_tests                       0x00163bd7 0x0 + 1457111
	46  browser_tests                       0x00160fee 0x0 + 1445870
	47  browser_tests                       0x00002916 0x0 + 10518
	48  ???                                 0x00000007 0x0 + 7

Jul 23, 2010

#5 thomasvl@chromium.org

(No comment was entered for this change.)

Labels: -Area-BuildTools Feature-Extensions ext

Jul 23, 2010

#6 thomasvl@chromium.org

Just for clarity - I think the extension issue might be what causes all this snowballing, but something within the test setup for both browser tests and ui tests is letting part of the spawned process tree be left around when things start going really slow.

Jul 23, 2010

#7 thomasvl@chromium.org

When test start backing up, the file save dialog is trying to save "download-test1.lib".  Sadly, there a are lot of tests that seem to use that file, so that isn't stopgap fix either.  A search in src/chrome gives::

./browser/cocoa/download_util_mac_unittest.mm:  FilePath testFile(FILE_PATH_LITERAL("download-test1.lib"));
./browser/download/download_uitest.cc:  FilePath file(FILE_PATH_LITERAL("download-test1.lib"));
./browser/download/download_uitest.cc:  FilePath file(FILE_PATH_LITERAL("download-test1.lib"));
./browser/download/download_uitest.cc:  FilePath file(FILE_PATH_LITERAL("download-test1.lib"));
./browser/download/download_uitest.cc:  FilePath file(FILE_PATH_LITERAL("download-test1.lib"));
./browser/download/download_uitest.cc:  FilePath file(FILE_PATH_LITERAL("download-test1.lib"));
./browser/download/download_uitest.cc:  FilePath file(FILE_PATH_LITERAL("download-test1.lib"));
./browser/download/download_uitest.cc:  FilePath file(FILE_PATH_LITERAL("download-test1.lib"));
./browser/download/download_uitest.cc:  FilePath file(FILE_PATH_LITERAL("download-test1.lib"));
./browser/extensions/file_reader_unittest.cc:  RunBasicTest("download-test1.lib");
./test/data/download_page1.html:  var download = "http://"+window.location.host+"/download-test1.lib";
./test/data/download_page3.html:  var download = "http://"+window.location.host+"/download-test1.lib";
./test/data/download_page4.html:	<form id="form" action="download-test1.lib" target="_blank">
./test/data/sync_xmlhttprequest_during_unload.html:        req.open('GET','download-test1.lib', false);

Jul 23, 2010

#8 thomasvl@chromium.org

http://crbug.com/42314 is already tracking the extension notifications test as flaky.

Jul 23, 2010

#9 bugdroid1@gmail.com

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=53512 

------------------------------------------------------------------------
r53512 | thomasvl@chromium.org | 2010-07-23 13:54:19 -0700 (Fri, 23 Jul 2010) | 4 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/extensions/notifications_apitest.cc?r1=53512&r2=53511

Disable a flaky test that seems to completely blow up with some frequency on the mac bots and seems to cause a ripple in hosing the bot in general.
BUG=42314,50060
TEST=Mac test bots stay greener
Review URL: http://codereview.chromium.org/2834069
------------------------------------------------------------------------

Jul 23, 2010

#10 thomasvl@chromium.org

The extensions notifications test is now disabled on Mac and I'm bouncing the bot again.  With the bot already in a bad state, I do see lots of save dialogs for that download-test1.lib come up.  So if this doesn't help things, trying to track down real mappings of tests invokes to those dialogs will help.  The problem is trying to figure out which ones really cause the dialog by mistake cause things cycle so quickly.

Jul 23, 2010

#11 bugdroid1@gmail.com

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=53547 

------------------------------------------------------------------------
r53547 | asargent@chromium.org | 2010-07-23 16:09:13 -0700 (Fri, 23 Jul 2010) | 11 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/download/download_uitest.cc?r1=53547&r2=53546
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/multipart_uitest.cc?r1=53547&r2=53546

Disable some failing ui tests on MacOS

This may have no impact on the real problem we're seeing on the bots since it 
may just kick the can down to make the next test fail, but I'm giving it a shot
anyway.

BUG=50060
TEST=none

TBR=thomasvl

------------------------------------------------------------------------

Jul 23, 2010

#12 asargent@chromium.org

 Issue 50097  has been merged into this issue.

Cc: thoma...@chromium.org andyb...@chromium.org s...@chromium.org stuartmo...@chromium.org asarg...@chromium.org

Jul 23, 2010

#13 bugdroid1@gmail.com

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=53553 

------------------------------------------------------------------------
r53553 | asargent@chromium.org | 2010-07-23 17:02:35 -0700 (Fri, 23 Jul 2010) | 7 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/multipart_uitest.cc?r1=53553&r2=53552

Fix a typo in previous attempt at disabling a test.

BUG=50060
TEST=none



------------------------------------------------------------------------

Jul 23, 2010

#14 nsylv...@google.com

A few tests were leaking processes all the time.  There were a few dozen of those leaked processes running on the machine when I looked.

chrome-bot 10087   0.0  0.7   554676  30892   ??  S     4:33PM   0:10.43 /b/slave/chromium-rel-mac6-tests/build/src/xcodebuild/Release/Chromium.app/Contents/MacOS/Chromium --no-first-run --no-default-browser-check --test-type=ui --user-data-dir=/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.wAtI23 --enable-file-cookies --testing-channel=ChromeTestingInterface:9755.39 --noerrdialogs --homepage=about:blank --disable-web-resources --metrics-recording-only --enable-logging --test-name=DownloadTest.FLAKY_CloseNewTab3 --allow-file-access-from-files --disable-tab-closeable-state-watcher
chrome-bot 10060   0.0  0.6   533540  23296   ??  S     4:33PM   0:10.25 /b/slave/chromium-rel-mac6-tests/build/src/xcodebuild/Release/Chromium.app/Contents/MacOS/Chromium --no-first-run --no-default-browser-check --test-type=ui --user-data-dir=/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.C7knP3 --enable-file-cookies --testing-channel=ChromeTestingInterface:9755.37 --noerrdialogs --homepage=about:blank --disable-web-resources --metrics-recording-only --enable-logging --test-name=DownloadTest.FLAKY_DontCloseNewTab3 --allow-file-access-from-files --disable-tab-closeable-state-watcher
chrome-bot 10040   0.0  0.5   532744  22876   ??  S     4:33PM   0:10.74 /b/slave/chromium-rel-mac6-tests/build/src/xcodebuild/Release/Chromium.app/Contents/MacOS/Chromium --no-first-run --no-default-browser-check --test-type=ui --user-data-dir=/var/folders/Dv/DvuOAgCiHqe-M55O-YhHlU+++TM/-Tmp-/.org.chromium.H6q5Wf --enable-file-cookies --testing-channel=ChromeTestingInterface:9755.35 --noerrdialogs --homepage=about:blank --disable-web-resources --metrics-recording-only --enable-logging --test-name=DownloadTest.FLAKY_CloseNewTab1 --allow-file-access-from-files --disable-tab-closeable-state-watcher



I disabled those tests.

Jul 27, 2010

#15 finnur@chromium.org

(No comment was entered for this change.)

Status: Assigned

Jul 29, 2010

#16 nirnimesh@chromium.org

This seems like a more general test infra problem right? Why are individual tests being disabled in that case?

I'm seeing my bots running pyauto tests (which uses the same underlying infra as ui_tests) hanging too.

Jul 29, 2010

#17 thomasvl@chromium.org

Yes, it's a general problem, but there are a few tests that seem to trigger it, so we're trying to tag those tests to help figure out what causes it and also to make sure we stop hosing bots while figuring it out.

If you are seeing it on pyauto tests, then that might be yet a different place where the launch/shutdown isn't handled write, we have some many different wrappers there are probably a few places with bugs like this that need people to dig into them.

Jul 29, 2010

#18 bauerb@chromium.org

(No comment was entered for this change.)

Cc: bau...@chromium.org

Aug 16, 2010

#19 phajdan.jr@chromium.org

Can we add a build step that kills all chrome processes? I think we have that for Windows, not sure about POSIX.

Aug 24, 2010

#20 phajdan.jr@chromium.org

Nicolas said offline that adding a step to kill all chrome processes sounds file, there was just no need to do so until now. I will test the change locally.

Aug 24, 2010

#21 thomasvl@chromium.org

FYI -

I got this to happen today on my local machine.  Something more then just orphaned processes also happens.  I've got an entry for Chromium stuck in my Dock, the dock things it's running, just not responding.  It won't force quit, and a ps listing doesn't find any matching process.

So adding a step to kill leaked ones will help, but when things go wrong, we seem to confuse things in other ways.  :(

Aug 25, 2010

#22 phajdan.jr@chromium.org

(No comment was entered for this change.)

Status: Available
Owner: ---
Cc: phajdan...@chromium.org

Aug 25, 2010

#23 andyb...@chromium.org

(No comment was entered for this change.)

Cc: -andyb...@chromium.org

Aug 31, 2010

#24 bugdroid1@gmail.com

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=58107 

------------------------------------------------------------------------
r58107 | tim@chromium.org | 2010-08-31 18:23:12 -0700 (Tue, 31 Aug 2010) | 6 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/download/download_uitest.cc?r1=58107&r2=58106

Disable DownloadTest.NewWindow everywhere.

BUG=44454, 50060
TBR=phajdan-jr

Review URL: http://codereview.chromium.org/3302001
------------------------------------------------------------------------

Summary: ui_test and browser_tests on mac test bots (10.5 & 10.6) seem to leak processes when things go wrong and take out the bot

Sep 1, 2010

#25 bugdroid1@gmail.com

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=58225 

------------------------------------------------------------------------
r58225 | phajdan.jr@chromium.org | 2010-09-01 12:57:25 -0700 (Wed, 01 Sep 2010) | 12 lines
Changed paths:
   M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/download/download_uitest.cc?r1=58225&r2=58224

GTTF: Disable all download tests, they are broken just so much.

They don't wait until the download finishes,
so a "do you want to close the browser, you have pending downloads?"
dialog appears, which makes buildbots leak a process and become hosed.

TBR=ahendrickson

BUG=35275, 48913, 50060
TEST=ui_tests

Review URL: http://codereview.chromium.org/3301006
------------------------------------------------------------------------

Summary: ui_test and browser_tests on mac test bots (10.5 &amp; 10.6) seem to leak processes when things go wrong and take out the bot

Sep 30, 2010

#26 bugdroid1@gmail.com

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=61036

------------------------------------------------------------------------
r61036 | phajdan.jr@chromium.org | Thu Sep 30 00:48:08 PDT 2010

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/download/download_uitest.cc?r1=61036&r2=61035&pathrev=61036

Undisable download UI tests. We shouldn't leak processes now if they fail.

Is flaky better than disabled? Dunno, but it looks better (tm).

BUG=35275, 48913, 50060
TEST=none

Review URL: http://codereview.chromium.org/3592001
------------------------------------------------------------------------

Nov 6, 2010

#27 phajdan.jr@chromium.org

I think the problem with leaking processes has been fixed. Assigning to self to remember to undisable the affected tests.

Status: Assigned
Owner: phajdan...@chromium.org
Labels: Tests-Disabled

Nov 30, 2010

#28 bugdroid1@gmail.com

The following revision refers to this bug:
    http://src.chromium.org/viewvc/chrome?view=rev&revision=67710

------------------------------------------------------------------------
r67710 | phajdan.jr@chromium.org | Tue Nov 30 07:10:16 PST 2010

Changed paths:
 M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/download/download_uitest.cc?r1=67710&r2=67709&pathrev=67710
 M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/history/multipart_uitest.cc?r1=67710&r2=67709&pathrev=67710
 M http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/extensions/notifications_apitest.cc?r1=67710&r2=67709&pathrev=67710

Undisable tests that were disabled because of the leaked processes issue.

TBR=thomasvl

BUG=50060
TEST=none

Review URL: http://codereview.chromium.org/5396003
------------------------------------------------------------------------

Dec 1, 2010

#29 phajdan.jr@chromium.org

(No comment was entered for this change.)

Status: Fixed

Dec 1, 2010

#30 bugdroid1@chromium.org

Verified label updated by AutoAllocator, contact AmolK or KrisR for details

Labels: Verifier-Rohitbm

Dec 14, 2010

#31 rohi...@chromium.org

(No comment was entered for this change.)

Status: Verified

Mar 18, 2011

#32 lafo...@chromium.org

Since getting back from vacation, I've already had to bounce 3 bots in ~ 24 hours for this problem.  What appears to be happening is this:

- Tests are failing to property shutdown the Chromium processes they startup, this leak of processes
- This builds up causing general slow downs causing more tests to fail (and more process to get leaked)
- Eventually the bot runs out of processes and even update steps fail

When I've logged in to check on things, the left over processes can have windows, I see the download confirmation window up (quitting with active downloads), I see the file save dialog asking to confirm saving the something like &quot;download.lib&quot;.

I'm gonna start digging through logs to see what causes it, but this is the initial state of info so other folks can find this.

Labels: -GreenTreeTaskForce bulkmove TaskForce-GreenTree

Oct 12, 2012

#33 bugdroid1@chromium.org

This issue has been closed for some time. No one will pay attention to new comments.
If you are seeing this bug or have new data, please click New Issue to start a new bug.

Labels: Restrict-AddIssueComment-Commit

Mar 10, 2013

#34 bugdroid1@chromium.org

(No comment was entered for this change.)

Labels: -Feature-Extensions -Tests-Disabled Cr-Platform-Extensions Cr-Tests-Disabled

► Sign in to add a comment