My favorites | Sign in
Project Home Downloads Wiki Issues Code Search
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 195911: Tests time out while attempting to revert DNS settings
4 people starred this issue and may be notified of changes. Back to list
 
Project Member Reported by derat@chromium.org, Mar 22, 2011
Tests sometimes fail with "Timed out waiting to revert DNS." errors[1].  This is coming from revert_dns() in client/cros/cros_ui_test.py, which is documented as attempting to clear any DNS settings and fetch new ones via DHCP.  Per cmasone, this may be caused by flimflam not responding in time due to being single-threaded.

1. http://build.chromium.org/p/chromiumos/builders/x86%20generic%20pre%20flight%20queue/builds/4235/steps/cbuildbot/logs/stdio
Mar 22, 2011
#1 achuith@chromium.org
Second bot closure today due to this
Labels: -Pri-2 Pri-1
Mar 25, 2011
#3 cmasone@chromium.org
Interestingly, dalecurtis says these failures do not occur in the test farm.  Also, tlambert told me the other day that he sees an 30 second gap in ALL system logs when these errors occur, which could indicate some kind of clock jump problem, or that the VMs don't get to run for 30+ seconds for some reason.
Mar 30, 2011
#4 cmasone@chromium.org
I think the real issue here is what tlambert was talking about, and I think this causes some of the otehr timeout problems we see too.  I just looked at and timeout-on-login failure and saw this same gap in the logs.  They all seemed to stop around 09:38:37, and nothing appeared in any of the logs until things timed out a bit over 30s later.

These failures have closed the tree multiple times in the last few days.  Terry, could you look at this, since you've already investigated somewhat?
Owner: tlambert...@gtempaccount.com
Cc: adlrchromiumorg@gmail.com
Labels: -Pri-1 -Area-Network -Sev-2 Pri-0 Area-Test Sev-1
Mar 30, 2011
#5 tlamb...@google.com
Sure.  Is this a live problem right now, or do we have a bounding interval for me to pick logs from?
Apr 28, 2011
#9 derat@chromium.org
For the sake of searching: This can also show up as "TimeoutError: Timed out waiting for DNS changes."
Status: Assigned
May 2, 2011
#10 cmas...@google.com
Issue chrome-os-partner:3563 has been merged into this issue.
Cc: kr...@google.com hynos...@google.com
May 18, 2011
#14 tlamb...@chromium.org
This most recent failure does not appear to be related with suspend/resume or other weird time related bugs.  This may be true of other failures as well.

The debug log indicates that the changes have successfully taken place, but have not been noticed by the python script:

5/18 12:00:37 DEBUG|cros_ui_te:0082| Considering eth0
05/18 12:00:37 DEBUG|cros_ui_te:0090| Stored 10.0.2.3 for eth0
05/18 12:00:37 DEBUG|cros_ui_te:0092| Using local DNS for eth0
05/18 12:00:40 DEBUG|miniFakeDn:0086| Response: www.google.com. -> 127.0.0.1
05/18 12:00:40 DEBUG|miniFakeDn:0086| Response: www.google.com. -> 127.0.0.1
05/18 12:00:40 DEBUG|miniFakeDn:0086| Response: www.google.com. -> 127.0.0.1
05/18 12:00:40 DEBUG|     httpd:0109| Translated path: /_/support/chrome/bin/topic/1142433/inproduct
05/18 12:00:40 ERROR|BaseHTTPSe:0447| localhost - - [18/May/2011 12:00:40] code 404, message File not found
05/18 12:00:40 ERROR|BaseHTTPSe:0447| localhost - - [18/May/2011 12:00:40] "GET /support/chrome/bin/topic/1142433/inproduct?hl=en-US HTTP/1.1" 404 -
05/18 12:00:40 DEBUG|     httpd:0120| URL /support/chrome/bin/topic/1142433/inproduct not in watch list
05/18 12:00:41 DEBUG|miniFakeDn:0086| Response: dl.google.com. -> 127.0.0.1
05/18 12:00:41 DEBUG|miniFakeDn:0086| Response: dl.google.com. -> 127.0.0.1
05/18 12:00:41 DEBUG|miniFakeDn:0086| Response: dl.google.com. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: mbokiszjsu. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: mygqsnsxdq. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: mbokiszjsu. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: mbokiszjsu. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: ihzesodusp. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: mygqsnsxdq. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: mygqsnsxdq. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: ihzesodusp. -> 127.0.0.1
05/18 12:00:42 DEBUG|miniFakeDn:0086| Response: ihzesodusp. -> 127.0.0.1
05/18 12:00:42 DEBUG|     httpd:0109| Translated path: /_
05/18 12:00:42 ERROR|BaseHTTPSe:0447| localhost - - [18/May/2011 12:00:42] code 404, message File not found
05/18 12:00:42 DEBUG|     httpd:0109| Translated path: /_
05/18 12:00:42 ERROR|BaseHTTPSe:0447| localhost - - [18/May/2011 12:00:42] "HEAD / HTTP/1.1" 404 -
05/18 12:00:42 ERROR|BaseHTTPSe:0447| localhost - - [18/May/2011 12:00:42] "HEAD / HTTP/1.1" 404 -localhost - - [18/May/2011 12:00:42] code 404, message File not found
05/18 12:00:42 DEBUG|     httpd:0109| Translated path: /_
05/18 12:00:42 ERROR|BaseHTTPSe:0447| localhost - - [18/May/2011 12:00:42] "HEAD / HTTP/1.1" 404 -
05/18 12:00:42 ERROR|BaseHTTPSe:0447| localhost - - [18/May/2011 12:00:42] "HEAD / HTTP/1.1" 404 -localhost - - [18/May/2011 12:00:42] code 404, message File not found
05/18 12:00:42 ERROR|BaseHTTPSe:0447| localhost - - [18/May/2011 12:00:42] "HEAD / HTTP/1.1" 404 -
05/18 12:00:47 ERROR|      test:0424| Exception escaping from test:
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 388, in _exec
    _cherry_pick_call(self.initialize, *args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 522, in _cherry_pick_call
    return func(*p_args, **p_dargs)
  File "/usr/local/autotest/tests/login_CryptohomeUnmounted/login_CryptohomeUnmounted.py", line 12, in initialize
    super(login_CryptohomeUnmounted, self).initialize(creds)
  File "/usr/local/autotest/cros/cros_ui_test.py", line 169, in initialize
    self.start_authserver()
  File "/usr/local/autotest/cros/cros_ui_test.py", line 132, in start_authserver
    self.use_local_dns()
  File "/usr/local/autotest/cros/cros_ui_test.py", line 97, in use_local_dns
    10)
  File "/usr/local/autotest/bin/site_utils.py", line 65, in poll_for_condition
    raise exception
TimeoutError: Timed out waiting for DNS changes.

=======

It may be worthwhile to add debug output both at the top of use_local_dns() and prior to the utils.poll_for_condition() loop.  It may also be worthwhile increasing the poll interval from .1S to 1S and add interior logging.

Cc: -adlrchromiumorg@gmail.com adlr@chromium.org
May 18, 2011
#15 bugdroid1@chromium.org
Commit: d7674f45f10d89a9a1baf6e65312663c9e2fe71b
 Email: cmasone@chromium.org

[autotest] Added logging inside cros_ui_test.__attempt_resolve

To try to suss out the DNS-related timeout issues during autotests, add some logging inside __attempt_resolve

BUG=chromium-os:13384
TEST=suite_Smoke -- ensure the tests pass, and that new log messages are seen

Change-Id: I267861ac62bdf2383a40bf60c8f47a44c09e62af
Reviewed-on: http://gerrit.chromium.org/gerrit/1154
Tested-by: Chris Masone <cmasone@chromium.org>
Reviewed-by: Chris Sosa <sosa@chromium.org>

M	client/cros/cros_ui_test.py
May 20, 2011
#17 tlamb...@chromium.org
Logs appear currently inaccessible to both myself and Chris, so we can't see the additional debug info right now.  Chris is pinging people on it.
May 20, 2011
#18 scottz@chromium.org
What logs are you trying to get at? 
Cc: sosa@chromium.org
May 20, 2011
#19 cmasone@chromium.org
The detailed autotest logs from the build in comment 16.  the CROS_ARCHIVE_URL doesn't seem to be right (it has a ... in the middle) and, even if it is...I could not find the test output there.
May 20, 2011
#20 scottz@chromium.org
Yeah looking at: https://sandbox.google.com/storage/?arg=chromeos-x86-mario/images/raw/_index.html#chromeos-x86-mario%2Frelease-master%2F...%2Fimages%2Fraw god only knows why there is a ... in there. I also do not see the logs.

Sosa is there some fix we can do quickly for the interim? If not I suggest we just sit tight until we move off of cbuild, cbuildbot handles this in a better way and we are very close to moving to it.
May 20, 2011
#21 sosa@chromium.org
Next time this happens ping me and I will grab the logs from the builder before they are cleaned up
May 20, 2011
#22 sosa@chromium.org
Speak of the devil it happened on mario pfq which we do have logs for.  Grab them here: https://sandbox.google.com/storage/chromeos-x86-mario/pre-flight-master/0.13.539.0-r6224cba9-b2577/_index.html
May 20, 2011
#24 cmasone@chromium.org
ah ha!

Here's a normal case:

05/20 12:36:23 DEBUG|cros_ui_te:0084| Considering eth0
05/20 12:36:23 DEBUG|cros_ui_te:0092| Stored 10.0.2.3 for eth0
05/20 12:36:23 DEBUG|cros_ui_te:0094| Using local DNS for eth0
05/20 12:36:23 DEBUG|cros_ui_te:0068| Resolve attempt for www.google.com got 74.125.224.83
05/20 12:36:24 DEBUG|miniFakeDn:0086| Response: www.google.com. -> 127.0.0.1
05/20 12:36:24 DEBUG|cros_ui_te:0068| Resolve attempt for www.google.com got 127.0.0.1

We call socket.gethostbyname once a second, and it returns localhost after a little bit, and we're good to go.

Here's a failure:

05/20 12:35:30 DEBUG|cros_ui_te:0084| Considering eth0
05/20 12:35:30 DEBUG|cros_ui_te:0092| Stored 10.0.2.3 for eth0
05/20 12:35:30 DEBUG|cros_ui_te:0094| Using local DNS for eth0
05/20 12:35:34 DEBUG|miniFakeDn:0086| Response: dl.google.com. -> 127.0.0.1
05/20 12:35:34 DEBUG|miniFakeDn:0086| Response: dl.google.com. -> 127.0.0.1
05/20 12:35:34 DEBUG|miniFakeDn:0086| Response: dl.google.com. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: wecabnzvzs. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: xysmvxwwrr. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: xysmvxwwrr. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: xysmvxwwrr. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: wecabnzvzs. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: wecabnzvzs. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: onrnklztnb. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: onrnklztnb. -> 127.0.0.1
05/20 12:35:35 DEBUG|miniFakeDn:0086| Response: onrnklztnb. -> 127.0.0.1
05/20 12:35:35 DEBUG|     httpd:0109| Translated path: /_
05/20 12:35:35 ERROR|BaseHTTPSe:0447| localhost - - [20/May/2011 12:35:35] code 404, message File not found
05/20 12:35:35 DEBUG|     httpd:0109| Translated path: /_
05/20 12:35:35 ERROR|BaseHTTPSe:0447| localhost - - [20/May/2011 12:35:35] "HEAD / HTTP/1.1" 404 -
05/20 12:35:35 ERROR|BaseHTTPSe:0447| localhost - - [20/May/2011 12:35:35] "HEAD / HTTP/1.1" 404 -localhost - - [20/May/2011 12:35:35] code 404, message File not found
05/20 12:35:35 DEBUG|     httpd:0109| Translated path: /_
05/20 12:35:35 ERROR|BaseHTTPSe:0447| localhost - - [20/May/2011 12:35:35] "HEAD / HTTP/1.1" 404 -
05/20 12:35:35 ERROR|BaseHTTPSe:0447| localhost - - [20/May/2011 12:35:35] "HEAD / HTTP/1.1" 404 -localhost - - [20/May/2011 12:35:35] code 404, message File not found
05/20 12:35:35 ERROR|BaseHTTPSe:0447| localhost - - [20/May/2011 12:35:35] "HEAD / HTTP/1.1" 404 -
05/20 12:35:40 DEBUG|cros_ui_te:0068| Resolve attempt for www.google.com got 74.125.224.81

For some reason, socket.gethostbyname doesn't return a result AT ALL until the 10s are up.  Unfortunately, I didn't add logging enough to tell me at which level this is failing...did we call it right away, and it sat around for 10s?  Did we not even call it for 10s?  I don't know.  But, this is useful data.  I will add more logging to, hopefully, illuminate this failure.

Taking ownership of this issue, as the problem is likely somewhere in the autotest goop, and I'm as good a person as any to take it on.
Owner: cmasone@chromium.org
May 20, 2011
#25 bugdroid1@chromium.org
Commit: d2b23550e910d1cab2a8e05dd60f0ab10f5bfba3
 Email: cmasone@chromium.org

[autotest] Continue adding logging to __attempt_resolve

We've seen that, in DNS-change failure cases, it seems that our attempt to resolve www.google.com doesn't even
get run until the timeout is hit, if at all.  Add more logging to try to suss out what's going on

BUG=chromium-os:13384
TEST=LoginSuccess

Change-Id: Ib5f2799e79cef3c9e4763291aeaba4581556fadc
Reviewed-on: http://gerrit.chromium.org/gerrit/1301
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Chris Masone <cmasone@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>

M	client/cros/cros_ui_test.py
May 24, 2011
#27 tlamb...@google.com

It's getting what it thinks is the wrong answer, and throwing the exception.  Since that's a valid IP address for the real site, this appears to be cached information; 10 seconds should have been enough time for it to settle, if this was a forced DNS server update.

I think this needs to go to Sam L. at this point, since it appears libc or flimflam related:

=====
05/23 11:21:01 DEBUG|cros_ui_te:0066| Attempting to resolve www.google.com to 127.0.0.1
05/23 11:21:01 DEBUG|miniFakeDn:0086| Response: nibqfjepqm. -> 127.0.0.1
05/23 11:21:01 DEBUG|     httpd:0109| Translated path: /_
05/23 11:21:01 ERROR|BaseHTTPSe:0447| localhost - - [23/May/2011 11:21:01] code 404, message File not found
05/23 11:21:01 ERROR|BaseHTTPSe:0447| localhost - - [23/May/2011 11:21:01] "HEAD / HTTP/1.1" 404 -
05/23 11:21:11 DEBUG|cros_ui_te:0069| Resolve attempt for www.google.com got 74.125.224.115
05/23 11:21:11 ERROR|      test:0424| Exception escaping from test:
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 388, in _exec
    _cherry_pick_call(self.initialize, *args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 522, in _cherry_pick_call
    return func(*p_args, **p_dargs)
  File "/usr/local/autotest/tests/login_LoginSuccess/login_LoginSuccess.py", line 44, in initialize
    super(login_LoginSuccess, self).initialize(creds)
  File "/usr/local/autotest/cros/cros_ui_test.py", line 172, in initialize
    self.start_authserver()
  File "/usr/local/autotest/tests/login_LoginSuccess/login_LoginSuccess.py", line 52, in start_authserver
    self.use_local_dns()
  File "/usr/local/autotest/cros/cros_ui_test.py", line 100, in use_local_dns
    timeout=10)
  File "/usr/local/autotest/bin/site_utils.py", line 65, in poll_for_condition
    raise exception
TimeoutError: Timed out waiting for DNS changes.
=====
May 24, 2011
#28 bugdroid1@chromium.org
Commit: 92b2cff98317a9b176a0ecc1d88079418846059b
 Email: cmasone@chromium.org

[autotest] Force FQDN when trying to DNS-resolve www.google.com in CrosUITest setup

In the setup phase of CrosUITest, we fake out DNS by re-routing all
DNS resolution requests to a local DNS server.  We check that this has
been done by attempting a DNS resolution of 'www.google.com' and
expecting to get back 127.0.0.1.  By forcing the use of a
fully-qualified-domain-name, we make the resolution attempt go
directly to the DNS server, bypassing any local potential sources of
failure or hanging.

BUG=chromium-os:13384
TEST=suite_Smoke

Change-Id: I58f0f7175db6f0e2001fe164c5935ef7e5a18e4d
Reviewed-on: http://gerrit.chromium.org/gerrit/1495
Reviewed-by: Chris Sosa <sosa@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>

M	client/cros/cros_ui_test.py
May 26, 2011
#30 cmasone@chromium.org
 Issue 15807  has been merged into this issue.
May 30, 2011
#31 cmasone@chromium.org
 Issue 15915  has been merged into this issue.
May 31, 2011
#34 dalecurtis@chromium.org
Another hit today:

http://chromeos-botmaster.mtv.corp.google.com:8026/builders/TOT%20Pre-Flight%20Queue/builds/2733/steps/cbuildbot/logs/stdio

Slightly different log this time:

05/31 12:06:02 DEBUG|cros_ui_te:0110| Considering eth0
05/31 12:06:02 DEBUG|cros_ui_te:0115| Reverted DNS for eth0
05/31 12:06:02 DEBUG|cros_ui_te:0066| Attempting to resolve www.google.com. to 127.0.0.1
05/31 12:06:12 ERROR|cros_ui_te:0072| [Errno -2] Name or service not known
May 31, 2011
#35 cmasone@chromium.org
I don't know that this  last report is the same thing.  Please file a new report with more log information (this snippet is too short for me to tell), or a link to the actual logs.  I'll dup it if it's the same.
Jun 1, 2011
#36 cmasone@chromium.org
After looking at more extensive logs, I am now convinced this is the same thing manifesting in a different way, due to the change from attempting to resolve "www.google.com" to "www.google.com."
Jun 1, 2011
#37 cmasone@chromium.org
(No comment was entered for this change.)
Labels: Iteration-31
Jun 1, 2011
#38 bugdroid1@chromium.org
Commit: d9cc4780baf3f53f966dec4ff84e37229c5144c5
 Email: cmasone@chromium.org

[autotest] Switch to getaddrinfo and print route table on DNS checks

gethostbyname is deprecated.  Switch to using getaddrinfo in the hopes of
getting better behavior.  Also, log the device routing table on every
DNS resolve request that occurs while we're trying to switch to/from
the local, fake, DNS server.  This info will hopefully help diagnose
the failures that will, in all likelihood, continue to occur even after
the switch to getaddrinfo.

BUG=chromium-os:13384
TEST=suite_Smoke

Change-Id: I07d1dc5a4e0292d6f0fecc5f3627db63bef9355e
Reviewed-on: http://gerrit.chromium.org/gerrit/1896
Reviewed-by: Chris Masone <cmasone@chromium.org>
Tested-by: Chris Masone <cmasone@chromium.org>

M	client/cros/cros_ui_test.py
Jun 1, 2011
#39 bugdroid1@chromium.org
Commit: 7b4284a072cd8c196e374b3c5bd4f8b890210c20
 Email: cmasone@chromium.org

[autotest] Implement gethostbyname by shelling out to ping and parsing output

Sigh.  Perhaps this will work more reliably than direct socket calls,
which hang sometimes :-/

BUG=chromium-os:13384
TEST=suite_Smoke

Change-Id: I9c1e9aa49c997667886d14198910ef3bedda9ceb
Reviewed-on: http://gerrit.chromium.org/gerrit/1931
Tested-by: Chris Masone <cmasone@chromium.org>
Reviewed-by: Darin Petkov <petkov@chromium.org>

M	client/cros/cros_ui_test.py
Jun 1, 2011
#40 bugdroid1@chromium.org
Commit: 9411b4fa7ef8e9bc93c6b4e36f7fa6fc731f1b7f
 Email: cmasone@chromium.org

[autotest] if ping times out, catch the error and allow us to try again

Now that we're forking ping and using that to check if we've successfully changed DNS servers, we need to ensure that we catch exceptions from that process and just retry.

BUG=chromium-os:13384
TEST=suite_Smoke

Change-Id: I5ebee22487776338f0230bbcd785effd5797a446
Reviewed-on: http://gerrit.chromium.org/gerrit/1939
Tested-by: Chris Masone <cmasone@chromium.org>
Reviewed-by: Darin Petkov <petkov@chromium.org>

M	client/cros/cros_ui_test.py
Jun 1, 2011
#41 bugdroid1@chromium.org
Commit: a216a188fb4c17e4c79f31b38e36f470f50d430c
 Email: cmasone@chromium.org

[autotest] Catch any exceptions coming from ping attempt so that we can retry

BUG=chromium-os:13384
TEST=suite_Smoke

Change-Id: I8e98c44db311eae63bacdfd4facbf1270fb99fe9
Reviewed-on: http://gerrit.chromium.org/gerrit/1951
Tested-by: Chris Masone <cmasone@chromium.org>
Reviewed-by: Ryan Cairns <rtc@chromium.org>

M	client/cros/cros_ui_test.py
Jun 2, 2011
#42 cmasone@chromium.org
Spoke to sleffler, and he's concerned that all I've done thus far is make it less likely that we'll hit a still-existing race condition.

For posterity, here is all the context I have:

For many of our autotests, we need to trick Chrome into talking to some locally running services.  The way we do this is by calling out to Flimflam over DBus and telling it to use a local, fake, DNS server for all interfaces it knows about.  This DNS server resolves everything to 127.0.0.1.  We issue the call and then, to be sure it took, we repeatedly attempt to resolve "www.google.com." until we get the answer we expect.  We do a similar thing when we switch the DNS server back to the real settings -- ask for the change, and then attempt resolves until we get an answer other than 127.0.0.1.

The bug is that, some percentage of the time, one of these resolve attempts will hang for many seconds.  This can happen on either transition (to the fake server, or back to the real server).  In a case where the failure happens on the real->fake transition, we'll see www.google.com still resolving to the real IP address for a couple of attempts and then a resolve attempt will hang.  In the good cases, we see several responses with the real IP followed by one with 127.0.0.1, and then we move on.

We used socket.gethostbyname() for a while, but tried getaddrinfo() for a bit yesterday because gethostbyname is supposedly deprecated.  With the former, we didn't see anything very interesting on a failure, but in the latter case, when the resolve attempt hung, the call would eventually fail with (-2, 'Name or service not known').

We've moved now to fork/exec'ing a ping attempt and then parsing its output.  My thought was that there was some kind of stale FD or something that we were blocking on, which would get cleared by the exec.  Sam's not convinced :-)
Jun 7, 2011
#43 cmasone@chromium.org
This seems like it's been worked around (hasn't repro'd since I landed the ping workarounds)

That said, we're still suspicious that there's an underlying thing.  sleffler will open a new bug for that.
Status: Verified
Sep 13, 2011
#44 chromeos...@chromium.org
(No comment was entered for this change.)
Labels: FixedIn-0.14-590.1 FixedIn-0.14-591.0
Oct 25, 2011
#45 chromeos...@chromium.org
(No comment was entered for this change.)
Labels: -FixedIn-0.14-590.1 -FixedIn-0.14-591.0 FixedIn-590.1.0 FixedIn-591.0.0
Jan 20, 2012
#46 chromeos...@chromium.org
(No comment was entered for this change.)
Labels: FixedInIndex-24e_1 FixedInIndex-14
Mar 6, 2013
#47 bugdroid1@chromium.org
(No comment was entered for this change.)
Labels: OS-Chrome
Mar 9, 2013
#48 bugdroid1@chromium.org
(No comment was entered for this change.)
Labels: -Area-Test -TreeCloser -Mstone-R12 Cr-Test Hotlist-TreeCloser M-12
Sign in to add a comment

Powered by Google Project Hosting