Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

external compression problem: file descriptors not closed causes hangs #132

Closed
GoogleCodeExporter opened this issue Mar 24, 2015 · 9 comments

Comments

@GoogleCodeExporter
Copy link

What steps will reproduce the problem?
1. download http://build.chromium.org/official/chromium-14.0.826.0.tar.bz2 
http://build.chromium.org/official/chromium-14.0.827.0.tar.bz2
2. xdelta3 -e -9 -s chromium-14.0.826.0.tar.bz2 chromium-14.0.827.0.tar.bz2 
chromium-14.0.826.0-14.0.827.0.tar.bz2.vcdiff

It will hang and not exit when done, I did a strace on it and it sits at:
close(8)                                = 0
close(9)                                = 0
close(3)                                = 0
waitpid(13093, 


What version of the product are you using? On what operating system?
xdelta 3.0.0 gcc 4.6.0 CRUX http://crux.nu

All the other deltas I have made have been fine. 
http://www.hvlinux.net/crux/distfiles/chromium/

Original issue reported on code.google.com by monster....@gmail.com on 22 Jul 2011 at 8:15

@GoogleCodeExporter
Copy link
Author

I have the same issue on a large MySQL database dump.  The problem is that not 
all the pipe file descriptors are closed by those processes not needing them.  
The parent xdelta3 closes (as you report from the strace) the pipe coming from 
the decompressor of the base source, but the decompressor does not exit because 
it still has more to write (which was not needed by the parent xdelta3), and 
because other related processes still hold the read side of that pipe open (not 
that they will ever use it.)

I haven't done this, but you might try the -f force option, as it seems to have 
some odd overloading with code that will drain the pipe coming in from the 
decompressor, even if that data is not needed.

What I did instead was to close some unused pipe fd's.  This enabled the 
close's you report above to get the last read descriptor open on the pipe from 
the decompressor, causing the decompressor (your pid 13093) to get a SIGPIPE 
and die, causing the above wait() to return and nothing to hang.

This fix unfortunately exposed a second confusion in the code.  The code thinks 
that any SIGPIPE is a serious problem to be reported, with recommendations of 
using -f and -F.  In this case of unneeded data from the source base, a SIGPIPE 
is not a problem, but rather would should happen.  The code now works, but 
after all is said and done and it has successfully written the desired result, 
it spits out the following annoying, misleading and disconcerting error message 
as it exits:
=============
xdelta3: external compression [pid 18527] signal 13
xdelta3: external compression commands failed
xdelta3: external compression closed the pipe
=============

Apply the attached patch to close some unused pipe fd's, and then ignore the 
above error message in this case:

I am using xdelta 3.0.0 on Kubuntu 11.04 (x86_64)



Original comment by ThePytho...@gmail.com on 18 Aug 2011 at 10:15

Attachments:

@GoogleCodeExporter
Copy link
Author

Well, one can hack xdelta3 to be less annoying when it decides to exit before 
reading all of the source base.  Attached is an update to my pjaa patch of the 
previous comment, with changes to the way xdelta3 reports SIGPIPE errors to be 
more gentle.

My changes are a bit of a hack, and not well thought out.  More work is needed 
on this code, but I do not expect to be the one doing it.

Original comment by ThePytho...@gmail.com on 19 Aug 2011 at 4:46

Attachments:

@GoogleCodeExporter
Copy link
Author

Issue 133 has been merged into this issue.

Original comment by josh.mac...@gmail.com on 18 Jun 2012 at 12:47

@GoogleCodeExporter
Copy link
Author

Great reporting!  Thank you and I will get this fixed.

Original comment by josh.mac...@gmail.com on 18 Jun 2012 at 12:48

  • Changed title: external compression problem: file descriptors not closed causes hangs
  • Changed state: Started

@GoogleCodeExporter
Copy link
Author

Issue 125 has a patch for this issue.  I'll get it into the next release.  
Thanks.

Original comment by josh.mac...@gmail.com on 30 Jun 2012 at 7:32

@GoogleCodeExporter
Copy link
Author

Issue 125 has been merged into this issue.

Original comment by josh.mac...@gmail.com on 30 Jun 2012 at 7:33

@GoogleCodeExporter
Copy link
Author

I started looking at this -- the "odd overloading" you mention jogged my memory 
of the last time I worked on it. I think the patch is probably correct, but I 
want to think about it some more. Thanks.

Original comment by josh.mac...@gmail.com on 9 Jul 2012 at 6:33

@GoogleCodeExporter
Copy link
Author

SVN 361


Original comment by josh.mac...@gmail.com on 9 Jul 2012 at 7:57

  • Changed state: Accepted

@GoogleCodeExporter
Copy link
Author

3.0.3 includes this fix.  Thank you!

Original comment by josh.mac...@gmail.com on 15 Jul 2012 at 10:30

  • Changed state: Fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant