My favorites ▼ | Sign in
Project Home Downloads Wiki Issues Code Search
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 205008: testUpdateKeepStateful and testUpdateWipeStateful fail w/ shflags error
1 person starred this issue and may be notified of changes. Back to list
 
Project Member Reported by dianders@chromium.org, Oct 21, 2011 
In the x86-mario canary (1154), we're getting a failure in testUpdateKeepStateful and testUpdateWipeStateful.

Errors:

  INFO: testUpdateWipeStateful: Performing a delta update from /b/cbuild/x86-mario-canary-master/src/scripts/latest_image/R16-1195.0.0-a1-b1153/chromiumos_qemu_image.bin to /b/cbuild/x86-mario-canary-master/src/build/images/x86-mario/latest-cbuildbot/chromiumos_test_image.bin in a VM
  EStopping the KVM instance
  
  ======================================================================
  ERROR: Tests if we can update normally.
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/b/cbuild/x86-mario-canary-master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_test.py", line 108, in testUpdateKeepStateful
      self.worker.PerformUpdate(self.target_image_path, self.base_image_path)
    File "/b/cbuild/x86-mario-canary-master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 146, in PerformUpdate
      key_to_use)
    File "/b/cbuild/x86-mario-canary-master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/vm_au_worker.py", line 64, in UpdateImage
      self.RunUpdateCmd(cmd, log_directory)
    File "/b/cbuild/x86-mario-canary-master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 218, in RunUpdateCmd
      raise update_exception.UpdateException(1, str(e))
  UpdateException: Command "['/b/cbuild/x86-mario-canary-master/src/scripts/lib/../bin/cros_run_vm_update', '--vm_image_path=/b/cbuild/x86-mario-canary-master/src/scripts/latest_image/R16-1195.0.0-a1-b1153/chromiumos_qemu_image.bin', '--update_log=/b/cbuild/x86-mario-canary-master/chroot/tmp/cbuildbotv3I4yf/test_harness/testUpdateKeepStateful/1_update/update_engine.log', '--snapshot', '--no_graphics', '--persist', '--kvm_pid=/tmp/kvm.9222', '--ssh_port=9222', '--stateful_update_flag=old', '--update_url=http://172.22.15.179:8080/update/cache/13232351d0e454d3695d969e66b486d0_db575495f9ec2967943ce2eca1bddbc1' failed.
  
  Output logged to /b/cbuild/x86-mario-canary-master/chroot/tmp/cbuildbotv3I4yf/test_harness/testUpdateKeepStateful/1_update/update.log
  
  ----------------------------------------------------------------------
  Ran 1 test in 202.558s
  
  FAILED (errors=1)
  EStopping the KVM instance
  
  ======================================================================
  ERROR: Tests if we can update after cleaning the stateful partition.
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/b/cbuild/x86-mario-canary-master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_test.py", line 128, in testUpdateWipeStateful
      'clean')
    File "/b/cbuild/x86-mario-canary-master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 146, in PerformUpdate
      key_to_use)
    File "/b/cbuild/x86-mario-canary-master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/vm_au_worker.py", line 64, in UpdateImage
      self.RunUpdateCmd(cmd, log_directory)
    File "/b/cbuild/x86-mario-canary-master/src/platform/crostestutils/au_test_harness/../../crostestutils/au_test_harness/au_worker.py", line 218, in RunUpdateCmd
      raise update_exception.UpdateException(1, str(e))
  UpdateException: Command "['/b/cbuild/x86-mario-canary-master/src/scripts/lib/../bin/cros_run_vm_update', '--vm_image_path=/b/cbuild/x86-mario-canary-master/src/scripts/latest_image/R16-1195.0.0-a1-b1153/chromiumos_qemu_image.bin', '--update_log=/b/cbuild/x86-mario-canary-master/chroot/tmp/cbuildbotv3I4yf/test_harness/testUpdateWipeStateful/1_update/update_engine.log', '--snapshot', '--no_graphics', '--persist', '--kvm_pid=/tmp/kvm.9223', '--ssh_port=9223', '--stateful_update_flag=clean', '--update_url=http://172.22.15.179:8080/update/cache/13232351d0e454d3695d969e66b486d0_db575495f9ec2967943ce2eca1bddbc1' failed.
  
  Output logged to /b/cbuild/x86-mario-canary-master/chroot/tmp/cbuildbotv3I4yf/test_harness/testUpdateWipeStateful/1_update/update.log
  
  ----------------------------------------------------------------------
  Ran 1 test in 189.141s
  
  FAILED (errors=1)

=====
=====

Digging through the build artifacts (test_results.gz => /./test_harness/testUpdateKeepStateful/1_update/update.log), this is the important bit:

  INFO    : Starting stateful update using URL http://172.22.15.179:8080/static/archive/cache/13232351d0e454d3695d969e66b486d0_db575495f9ec2967943ce2eca1bddbc1
  .: 10: Can't open /usr/share/misc/shflags
  ERROR   : Stateful update was not successful.

=====
=====

Apparently, shflags moved recently.  Need to track down what bit got missed.  ...also, figure out why other canaries didn't hit this?
Oct 21, 2011
#1 dianders@chromium.org 
(No comment was entered for this change.)
Owner: sonnyrao@chromium.org
Labels: -Pri-2 -Sev-1 Pri-1 Sev-2
Oct 21, 2011
#2 sonnyrao@chromium.org 
Well shflags didn't exactly move, we've actually kept both copies around during the transition period.  This failure is basicaly where /usr/bin/stateful_update gets copied from the chroot (package is cros-devutils) to the target during an update and then gets run on the target.  The target *should* have had both /usr/lib/shflags and /usr/share/misc/shflags on it.  I'm downloading the build artifacts and going to examine the image.
Oct 21, 2011
#3 sonnyrao@chromium.org 
well, i checked both chromiumos_test_image.bin and chromiumos_qemu_image.bin and both images contained both /usr/lib/shflags and /usr/share/misc/shflags as expected... so I don't really have an explanation for this failure
Oct 21, 2011
#4 sonnyrao@chromium.org 
Ok, after more careful examination of the logs, it looks like the stateful_update script actually runs in the *old* image, and it looks like the mario builder hasn't updated it's image in several days and didn't have the new version of shflags installed yet, while the other canary builders have built in the meantime and they do have the new version.  

So, in order to make the mario canary pass and also to help other developers who may be updating older images I'll update the script to check both locations
Oct 21, 2011
#5 sonnyrao@chromium.org 
(No comment was entered for this change.)
Status: Started
Labels: Iteration-41
Oct 21, 2011
#6 bugdroid1@chromium.org 
Commit: 7e44ca25e7b4945cc1a57f2d49273ef004fb9446
 Email: sonnyrao@chromium.org

Check for both new and old shflags during stateful update

BUG=chromium-os:21966
TEST=perform an update from an old build that doesn't have dev-util/shflags
installed to a current tot build

Change-Id: I999e49f658b99302b591450bf3c3788181408a91
Reviewed-on: http://gerrit.chromium.org/gerrit/10533
Reviewed-by: Doug Anderson <dianders@chromium.org>
Tested-by: Sonny Rao <sonnyrao@chromium.org>

M	stateful_update
Oct 21, 2011
#7 sonnyrao@chromium.org 
(No comment was entered for this change.)
Status: Fixed
Nov 7, 2011
#8 kr...@chromium.org 
(No comment was entered for this change.)
Status: Verified
Jan 9, 2012
#9 chromeos...@chromium.org 
(No comment was entered for this change.)
Labels: FixedIn-1212.0.0
Jan 20, 2012
#10 chromeos...@chromium.org 
(No comment was entered for this change.)
Labels: FixedInIndex-29
Mar 6, 2013
#11 lafo...@google.com 
(No comment was entered for this change.)
Labels: OS-Chrome
Mar 9, 2013
#12 bugdroid1@chromium.org 
(No comment was entered for this change.)
Labels: -Area-Platform -TreeCloser Cr-OS-Kernel Hotlist-TreeCloser
► Sign in to add a comment

Powered by Google Project Hosting