Issue 125 - boto - Failure to handle invalid chunked HTTP response from S3

Issue 125:	Failure to handle invalid chunked HTTP response from S3
	1 person starred this issue and may be notified of changes.	Back to list

Status:	Fixed
Owner:	----
Closed:	Nov 2008
Type-Defect
Priority-Medium
Component-S3

Reported by pjdorrell, Jun 27, 2008

I intermittently get the following stacktrace (see attachment for full trace):

  File
"c:\python25\lib\site-packages\boto-1.3a-py2.5.egg\boto\s3\bucket.py", line
99, in lookup
    return self.get_key(key_name)
  File
"c:\python25\lib\site-packages\boto-1.3a-py2.5.egg\boto\s3\bucket.py", line
113, in get_key
    response = self.connection.make_request('HEAD', self.name, key_name)
  File
"c:\python25\lib\site-packages\boto-1.3a-py2.5.egg\boto\s3\connection.py",
line 219, in make_request
    data, host, auth_path, sender)
  File
"c:\python25\lib\site-packages\boto-1.3a-py2.5.egg\boto\connection.py",
line 304, in make_request
    return self._mexe(method, path, data, headers, host, sender)
  File
"c:\python25\lib\site-packages\boto-1.3a-py2.5.egg\boto\connection.py",
line 253, in _mexe
    body = response.read()
  File "c:\Python25\lib\httplib.py", line 509, in read
    return self._read_chunked(amt)
  File "c:\Python25\lib\httplib.py", line 548, in _read_chunked
    chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: ''

This is running boto1.3a release on Python 2.5.1 on Windows Vista.

Looking at the stack dump, several things are going on:

1. Amazon S3 has returned a 500 or 503 status, indicating a server error.
2. Amazon S3 returns the body as an invalid "chunked" HTTP response (which
may be completely empty).
3. Python httplib.py does not correctly handle the invalid response.
Instead it throws a ValueError.
4. Boto does not handle this error thrown from httplib.py.

Looking at newer Python versions, I can see that:

1. Python 2.5.2 contains the same error in httplib.py
2. Python 2.6b1 contains code to catch _any_ formatting error when reading
the chunk size, and throw an IncompleteRead (even this fix is less than
ideal, because if your server application is mis-formatting HTTP chunk size
lines, then the client-side error is going to be uninformative).

An ideal solution would be for the httplib.py code to be fixed in all
versions of Python to return an appropriate HttpException in all cases, and
the existing Boto code already handles HttpException's correctly.

A more pragmatic solution in the short-term would be to catch and swallow
all ValueError exceptions that occur when reading the HTTP response body
for a server error 500 or 503 response (since the content is presumably
irrelevant anyway, all you are going to do is try again or fail completely).

	ChunkedHttpResponseError.log
	1.9 KB Download

Comment 1 by pjdorrell, Jun 28, 2008

The related Python bug is 900744 at http://bugs.python.org/issue900744

Comment 2 by lbates35406, Jul 13, 2008

Not a "permanent" solution but something like the following at least "retries":

#
# Wrapping this in try: block because of unfixed errror in
# boto/httplib.  For sanity, I'm only going to try 50 times
#
success = False
n = 0
while not success and n < 50:
    n += 1
    try:
        #
        # Read meta-data from S3 to get k.last_modified
        #
        k = self.bucket.get_key(S3path)

    except ValueError:
        time.sleep(n) # Sleep progressively longer
        continue

    else:
        success = True

Comment 4 by jesse.lovelace, Jul 18, 2008

Can this be incorporated into boto at a lower level?  Using this above boto seems to cause a great deal of code 
bloat--I get this error somewhat frequently from s3.

Comment 5 by Mitch.Garnaat, Aug 30, 2008

You can inform boto to catch all ValueError exceptions by adding that exception to
the http_exceptions attribute on the S3Connection object.  Actually, it looks like
that value is currently a tuple so you would have to create a new tuple containing
the old values and the ValueError.  I'll change that to be a list.

I haven't done enough research to know whether it makes sense to just add ValueError
to that list as the default.  I don't really see an issue with that but I'd have to
do more testing before committing that.

Labels: Component-S3

Comment 6 by Mitch.Garnaat, Nov 20, 2008

I believe this is not fixed in r928 and r936

Status: Fixed