urllib3 - Project Hosting on Google Code

Today 2 hours ago issue 6 (Doesn't properly recycle closed HTTPConnections) Status changed by shazow - I felt that was a reasonable solution, but I suppose checking if the socket is open before making the request is wise. Is this what you had in mind or something else? Status: Accepted I felt that was a reasonable solution, but I suppose checking if the socket is open before making the request is wise. Is this what you had in mind or something else? Status: Accepted Last 7 days Dec 15, 2009 issue 6 (Doesn't properly recycle closed HTTPConnections) reported by ionel.mc - There is no handling for closed HTTPConnection in the queue. The library would rather erroneously eat up a retry (via a HTTPConnection exception) in that case. There is no handling for closed HTTPConnection in the queue. The library would rather erroneously eat up a retry (via a HTTPConnection exception) in that case. Dec 15, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rapsys_e...@yahoo.com - in my code i tried modifying the values of redirect (e.g. no value, true, false): {{{ # Make the request and capture the response try: request = self.vt_http.post_url('url_here', post_data, redirect=False) except urllib3.HTTPError, e: print "Error: " + e.strerror }}} also added this line inside connectionpool: {{{ print "URL1 %s\n" % response.headers.get('location') # Handle redirection if redirect and response.status in [301, 302, 303, 307] and 'location' in response.headers: # Redirect, retry print "URL2 %s\n" % response.headers.get('location') log.info("Redirecting %s -> %s" % (url, response.headers.get('location'))) return self.urlopen(method, response.headers.get('location'), body, headers, retries-1, redirect) return response }}} in my code if redirect is False, both URL1 and URL2 are not getting the correct URL if redirect is True, both URL1 and URL2 gets the correct URL but since redirect is true, another self.urlopen(...) will be called and follows the redirect im a newbie to this, but what i need is redirect=False, get url referrer and don't follow redirection... so that in my code, i can use the request.headers.get('location') and get the referrer... hope this makes sense... in my code i tried modifying the values of redirect (e.g. no value, true, false): {{{ # Make the request and capture the response try: request = self.vt_http.post_url('url_here', post_data, redirect=False) except urllib3.HTTPError, e: print "Error: " + e.strerror }}} also added this line inside connectionpool: {{{ print "URL1 %s\n" % response.headers.get('location') # Handle redirection if redirect and response.status in [301, 302, 303, 307] and 'location' in response.headers: # Redirect, retry print "URL2 %s\n" % response.headers.get('location') log.info("Redirecting %s -> %s" % (url, response.headers.get('location'))) return self.urlopen(method, response.headers.get('location'), body, headers, retries-1, redirect) return response }}} in my code if redirect is False, both URL1 and URL2 are not getting the correct URL if redirect is True, both URL1 and URL2 gets the correct URL but since redirect is true, another self.urlopen(...) will be called and follows the redirect im a newbie to this, but what i need is redirect=False, get url referrer and don't follow redirection... so that in my code, i can use the request.headers.get('location') and get the referrer... hope this makes sense... Dec 15, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by mackst...@gmail.com - If you implement the pool of pools, I nominate that it be called HTTPOcean If you implement the pool of pools, I nominate that it be called HTTPOcean Dec 15, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by mackst...@gmail.com - So for crawling over multiple hosts, I'd want a connection pool for each host. I would then probably want to have a "pool of pools" that is a mapping between hostnames and pool objects, with some sort of automatic expiring of pools that haven't been used in a while. Are there any plans to roll this functionality into urllib3? Also, while urllib3 has the advantage of thread safety (and maybe pipelining, not sure) over httplib2, httplib2 has the advantage of caching. Are there any plans to implement caching, or somehow merge these two projects together in some fashion? So for crawling over multiple hosts, I'd want a connection pool for each host. I would then probably want to have a "pool of pools" that is a mapping between hostnames and pool objects, with some sort of automatic expiring of pools that haven't been used in a while. Are there any plans to roll this functionality into urllib3? Also, while urllib3 has the advantage of thread safety (and maybe pipelining, not sure) over httplib2, httplib2 has the advantage of caching. Are there any plans to implement caching, or somehow merge these two projects together in some fashion? Dec 15, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rapsys_e...@yahoo.com - how do you get the last url accessed using urllib3? e.g. using urllib2 {{{ encoded_data = urllib.urlencode(post_data) request = urllib2.Request(url, encoded_data) # Make the request and capture the response try: response = urllib2.urlopen(request) except urllib2.URLError, e: print "Error: " + e.strerror return False result = response.geturl() }}} how do you get the last url accessed using urllib3? e.g. using urllib2 {{{ encoded_data = urllib.urlencode(post_data) request = urllib2.Request(url, encoded_data) # Make the request and capture the response try: response = urllib2.urlopen(request) except urllib2.URLError, e: print "Error: " + e.strerror return False result = response.geturl() }}} Dec 15, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rapsys_e...@yahoo.com - how do you get the last url accessed using urllib3? <BR>e.g. using urllib2 <BR>encoded_data = urllib.urlencode(post_data) <BR>request = urllib2.Request(url, encoded_data) <BR># Make the request and capture the response <BR>try: <BR> response = urllib2.urlopen(request) <BR>except urllib2.URLError, e: <BR> print "Error: " + e.strerror <BR> return False <BR>result = response.geturl() how do you get the last url accessed using urllib3? <BR>e.g. using urllib2 <BR>encoded_data = urllib.urlencode(post_data) <BR>request = urllib2.Request(url, encoded_data) <BR># Make the request and capture the response <BR>try: <BR> response = urllib2.urlopen(request) <BR>except urllib2.URLError, e: <BR> print "Error: " + e.strerror <BR> return False <BR>result = response.geturl() Dec 15, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rapsys_e...@yahoo.com - how do you get the last url used using urllib3? e.g. from urllib2 encoded_data = urllib.urlencode(post_data) request = urllib2.Request(url, encoded_data) # Make the request and capture the response try: response = urllib2.urlopen(request) except urllib2.URLError, e: print "Error: " + e.strerror return False result = response.geturl() print "URL from redirect %s" % result how do you get the last url used using urllib3? e.g. from urllib2 encoded_data = urllib.urlencode(post_data) request = urllib2.Request(url, encoded_data) # Make the request and capture the response try: response = urllib2.urlopen(request) except urllib2.URLError, e: print "Error: " + e.strerror return False result = response.geturl() print "URL from redirect %s" % result Dec 15, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by wolf550e - How do you integrate this with [http://code.google.com/p/httplib2/ httplib2]? How do you integrate this with [http://code.google.com/p/httplib2/ httplib2]? Last 30 days Dec 10, 2009 issue 5 (https not supported) Status changed by shazow - Done in 0.3! Status: Fixed Done in 0.3! Status: Fixed Dec 10, 2009 issue 1 (Port is not optional (but should be)) Status changed by shazow - Status: Fixed Status: Fixed Dec 10, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by shazow - Most problems were fixed in the new release (not including the logger warning, not sure if that's a good idea considering it's supposed to be used in conjunction with other codebases... hrm.) Also, now with HTTPS support, wee! Most problems were fixed in the new release (not including the logger warning, not sure if that's a good idea considering it's supposed to be used in conjunction with other codebases... hrm.) Also, now with HTTPS support, wee! Dec 10, 2009 Examples (HTTPConnectionPool code examples) Wiki page edited by shazow Dec 10, 2009 3 Wiki pages changed by shazow - Added Examples Deleted MassDownloader, WorkerpoolDesignPatterns Added Examples Deleted MassDownloader, WorkerpoolDesignPatterns Dec 10, 2009 3 Wiki pages changed by shazow - Added MassDownloader, WorkerpoolDesignPatterns Deleted Examples Added MassDownloader, WorkerpoolDesignPatterns Deleted Examples Dec 10, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by shazow - Ok, new release is sent, but I can't update this wiki page with new examples, Google keeps giving me a 500 error. Drat. Ok, new release is sent, but I can't update this wiki page with new examples, Google keeps giving me a 500 error. Drat. Dec 10, 2009 urllib3-0.3.tar.gz (Added HTTPS support, fixed couple of bugs, refactored (break...) file uploaded by shazow - Labels: Featured Type-Source OpSys-All Labels: Featured Type-Source OpSys-All Dec 10, 2009 Revision 83109c65b1 (Ignore dist) pushed by shazow - Ignore dist Ignore dist Dec 10, 2009 Revision ac7d1013e7 (Updated README and version bump) pushed by shazow - Updated README and version bump Updated README and version bump Dec 10, 2009 urllib3-0.3.tar.gz (Added HTTPS support, fixed couple of bugs, refactored (break...) file uploaded by shazow - Labels: Type-Source OpSys-All Featured Labels: Type-Source OpSys-All Featured Dec 10, 2009 4 new revisions pushed by shazow - 2e1937e668:Adding ignore file 2046e0065a:Applied victor.vde's patch 5bcf5597eb:Cleaned up the previous patch f07151e113:Refactored HTTPS code, added unit tests 2e1937e668:Adding ignore file 2046e0065a:Applied victor.vde's patch 5bcf5597eb:Cleaned up the previous patch f07151e113:Refactored HTTPS code, added unit tests Dec 10, 2009 Revision edac42f631 (Initial import of source.) pushed by shazow - Initial import of source. Initial import of source. Dec 10, 2009 issue 3 (No improvement on python 2.4) Status changed by shazow - Hi Dan, the improvement is fundamental in the different approaches between pooling and not. If you scale this to large number of requests and busy servers, you'll definitely see improvements, no doubt about it. Status: Done Hi Dan, the improvement is fundamental in the different approaches between pooling and not. If you scale this to large number of requests and busy servers, you'll definitely see improvements, no doubt about it. Status: Done Dec 10, 2009 issue 5 (https not supported) changed by shazow - Hi Victor, thanks so much for the patch! I promise promise promise I'll do a new release this weekend, including https (I already have a lot of code for it, most of it is what you've done plus a few bonuses). Ahhh! :-) Thanks again. Status: Started Labels: Priority-High −Priority-Medium Hi Victor, thanks so much for the patch! I promise promise promise I'll do a new release this weekend, including https (I already have a lot of code for it, most of it is what you've done plus a few bonuses). Ahhh! :-) Thanks again. Status: Started Labels: Priority-High −Priority-Medium Dec 10, 2009 issue 5 (https not supported) commented on by victor.vde - Fixed patch Fixed patch Dec 10, 2009 issue 5 (https not supported) reported by victor.vde - urllib3 always uses a HTTPConnection, but it should support HTTPSConnection as well. Patch against the current SVN attached. urllib3 always uses a HTTPConnection, but it should support HTTPSConnection as well. Patch against the current SVN attached. Dec 04, 2009 issue 2 (Logging error if no Content-length header) Status changed by shazow - Sorry for the late response, I've been distracted with many other projects. I believe this has been fixed in trunk for some time now, but still pending a new release. Status: Fixed Sorry for the late response, I've been distracted with many other projects. I believe this has been fixed in trunk for some time now, but still pending a new release. Status: Fixed Dec 04, 2009 issue 4 (Pool init fails if Port is undefined) Status changed by shazow - This is fixed in trunk, hopefully a new release will come once I get some significant changes in. Status: Fixed This is fixed in trunk, hopefully a new release will come once I get some significant changes in. Status: Fixed Dec 04, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by shazow - Good idea, I should do another release soon. Anyone interested in joining and helping maintain? Good idea, I should do another release soon. Anyone interested in joining and helping maintain? Nov 30, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by david.ascher - There should probably be a default logger registered: No handlers could be found for logger "urllib3.connectionpool" isn't a great error message =) There should probably be a default logger registered: No handlers could be found for logger "urllib3.connectionpool" isn't a great error message =) Earlier this year Sep 14, 2009 issue 4 (Pool init fails if Port is undefined) reported by ambrose.ncube - HTTPConnectionPool.__init__ fails if the url port is undefined def __init__(self, host, port=None, timeout=None, maxsize=10): #.... self.port = int(port if port is not None else 80 ) # init fails otherwise if port number is not defined/None(default) HTTPConnectionPool.__init__ fails if the url port is undefined def __init__(self, host, port=None, timeout=None, maxsize=10): #.... self.port = int(port if port is not None else 80 ) # init fails otherwise if port number is not defined/None(default) Sep 14, 2009 issue 4 (Pool init fails if Port is undefined) reported by ambrose.ncube - HTTPConnectionPool.__init__ fails if the url port is undefined def __init__(self, host, port=None, timeout=None, maxsize=10): #.... self.port = int(port if port is not None else 80 ) # init fails otherwise if port number is not defined/None(default) HTTPConnectionPool.__init__ fails if the url port is undefined def __init__(self, host, port=None, timeout=None, maxsize=10): #.... self.port = int(port if port is not None else 80 ) # init fails otherwise if port number is not defined/None(default) Jul 14, 2009 issue 3 (No improvement on python 2.4) reported by danle...@gmail.com - What steps will reproduce the problem? 1. Run the benchmark.py (alternatively, switch the order of urllib_get and pool_get) What is the expected output? What do you see instead? Faster result with the pooling What version of the product are you using? On what operating system? Urllib 3.02 Python 2.4 Windows Please provide any additional information below. On 15 requests, I get similar result with or without pooling What steps will reproduce the problem? 1. Run the benchmark.py (alternatively, switch the order of urllib_get and pool_get) What is the expected output? What do you see instead? Faster result with the pooling What version of the product are you using? On what operating system? Urllib 3.02 Python 2.4 Windows Please provide any additional information below. On 15 requests, I get similar result with or without pooling Jul 02, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rpg...@hotmail.com - in example : {{{ In [25]: print r.status, r.length, len(r.data) 200--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) C:\Users\Kan\<ipython console> in <module>() AttributeError: 'HTTPResponse' object has no attribute 'length' }}} have change to : {{{ In [26]: print r.status, len(r.data) 200 83 }}} in example : {{{ In [25]: print r.status, r.length, len(r.data) 200--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) C:\Users\Kan\<ipython console> in <module>() AttributeError: 'HTTPResponse' object has no attribute 'length' }}} have change to : {{{ In [26]: print r.status, len(r.data) 200 83 }}} Jul 02, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rpg...@hotmail.com - {{{ In [6]: http_pool = HTTPConnectionPool('google.com') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) C:\Users\Kan\<ipython console> in <module>() C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n __init__(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError: int() argument must be a string or a number, not 'NoneType' }}} I change to : {{{ In [7]: http_pool = HTTPConnectionPool('google.com', port='80') In [8]: }}} {{{ In [6]: http_pool = HTTPConnectionPool('google.com') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) C:\Users\Kan\<ipython console> in <module>() C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n __init__(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError: int() argument must be a string or a number, not 'NoneType' }}} I change to : {{{ In [7]: http_pool = HTTPConnectionPool('google.com', port='80') In [8]: }}} Jul 02, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rpg...@hotmail.com - `inline code` `In [6]: http_pool = HTTPConnectionPool('google.com')` `---------------------------------------------------------------------------` `TypeError Traceback (most recent call last)` `` `C:\Users\Kan\<ipython console> in <module>()` `` `C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc in __init__(self, host, port, timeout, maxsize)` ` 81 self.pool = Queue(maxsize)` ` 82 self.host = host` `---> 83 self.port = int(port)` ` 84 self.timeout = timeout` ` 85 self.num_connections = count()` `` `TypeError: int() argument must be a string or a number, not 'NoneType'` `` `In [7]: http_pool = HTTPConnectionPool('google.com', port='80')` `inline code` `In [6]: http_pool = HTTPConnectionPool('google.com')` `---------------------------------------------------------------------------` `TypeError Traceback (most recent call last)` `` `C:\Users\Kan\<ipython console> in <module>()` `` `C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc in __init__(self, host, port, timeout, maxsize)` ` 81 self.pool = Queue(maxsize)` ` 82 self.host = host` `---> 83 self.port = int(port)` ` 84 self.timeout = timeout` ` 85 self.num_connections = count()` `` `TypeError: int() argument must be a string or a number, not 'NoneType'` `` `In [7]: http_pool = HTTPConnectionPool('google.com', port='80')` Jul 02, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rpg...@hotmail.com - ` In [6]: http_pool = HTTPConnectionPool('google.com') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) C:\Users\Kan\<ipython console> in <module>() C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n __init__(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError: int() argument must be a string or a number, not 'NoneType' In [7]: http_pool = HTTPConnectionPool('google.com', port'80') ` ` In [6]: http_pool = HTTPConnectionPool('google.com') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) C:\Users\Kan\<ipython console> in <module>() C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n __init__(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError: int() argument must be a string or a number, not 'NoneType' In [7]: http_pool = HTTPConnectionPool('google.com', port'80') ` Jul 02, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rpg...@hotmail.com - `inline code` In [6]: http_pool = HTTPConnectionPool('google.com') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) C:\Users\Kan\<ipython console> in <module>() C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n __init__(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError: int() argument must be a string or a number, not 'NoneType' In [7]: http_pool = HTTPConnectionPool('google.com', port'80') `inline code` `inline code` In [6]: http_pool = HTTPConnectionPool('google.com') --------------------------------------------------------------------------- TypeError Traceback (most recent call last) C:\Users\Kan\<ipython console> in <module>() C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n __init__(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError: int() argument must be a string or a number, not 'NoneType' In [7]: http_pool = HTTPConnectionPool('google.com', port'80') `inline code` Jul 02, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rpg...@hotmail.com - in The Basics by example have error from code :<br/> <br/> http_pool = HTTPConnectionPool('google.com')<br/> <br/> error detail :<br/> <br/> C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n init(self, host, port, timeout, maxsize)<br/> 81 self.pool = Queue(maxsize)<br/> 82 self.host = host ---><br/> 83 self.port = int(port)<br/> 84 self.timeout = timeout<br/> 85 self.num_connections = count()<br/> <br/> TypeError??: int() argument must be a string or a number, not 'NoneType??'<br/> <br/> change to :<br/> <br/> http_pool = HTTPConnectionPool('google.com', port='80')<br/> thank you in The Basics by example have error from code :<br/> <br/> http_pool = HTTPConnectionPool('google.com')<br/> <br/> error detail :<br/> <br/> C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n init(self, host, port, timeout, maxsize)<br/> 81 self.pool = Queue(maxsize)<br/> 82 self.host = host ---><br/> 83 self.port = int(port)<br/> 84 self.timeout = timeout<br/> 85 self.num_connections = count()<br/> <br/> TypeError??: int() argument must be a string or a number, not 'NoneType??'<br/> <br/> change to :<br/> <br/> http_pool = HTTPConnectionPool('google.com', port='80')<br/> thank you Jul 02, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rpg...@hotmail.com - in The Basics by example have error from code : http_pool = HTTPConnectionPool('google.com') error detail : C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n init(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError?: int() argument must be a string or a number, not 'NoneType?' change to : http_pool = HTTPConnectionPool('google.com', port='80') in The Basics by example have error from code : http_pool = HTTPConnectionPool('google.com') error detail : C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n init(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError?: int() argument must be a string or a number, not 'NoneType?' change to : http_pool = HTTPConnectionPool('google.com', port='80') Jul 02, 2009 Examples (HTTPConnectionPool code examples) Wiki page commented on by rpg...@hotmail.com - in The Basics by example have error from code : <code> http_pool = HTTPConnectionPool('google.com') <code> error detail : <code> C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n __init__(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError: int() argument must be a string or a number, not 'NoneType' </code> change to : http_pool = HTTPConnectionPool('google.com', port='80') in The Basics by example have error from code : <code> http_pool = HTTPConnectionPool('google.com') <code> error detail : <code> C:\Python25\lib\site-packages\urllib3-0.2-py2.5.egg\urllib3\connectionpool.pyc i n __init__(self, host, port, timeout, maxsize) 81 self.pool = Queue(maxsize) 82 self.host = host ---> 83 self.port = int(port) 84 self.timeout = timeout 85 self.num_connections = count() TypeError: int() argument must be a string or a number, not 'NoneType' </code> change to : http_pool = HTTPConnectionPool('google.com', port='80') Jun 03, 2009 r26 (Fixed bug in debug log) committed by shazow - Fixed bug in debug log Fixed bug in debug log May 21, 2009 Examples (HTTPConnectionPool code examples) Wiki page edited by shazow Feb 28, 2009 issue 2 (Logging error if no Content-length header) commented on by savant4u - Hi, I am also facing similar issue. i download urllib3 and then try to run # python benchmark.py I got following exception .. i am using python 2.5 on Fedora 10 Running pool_get ... Traceback (most recent call last): File "benchmark.py", line 52, in <module> pool_get(TO_DOWNLOAD) File "benchmark.py", line 42, in pool_get pool = HTTPConnectionPool.from_url(url_list[0]) File "/usr/lib/python2.5/site-packages/urllib3-0.2-py2.5.egg/urllib3/connectionpool.py", line 120, in from_url return HTTPConnectionPool(host, port=port, timeout=timeout, maxsize=maxsize) File "/usr/lib/python2.5/site-packages/urllib3-0.2-py2.5.egg/urllib3/connectionpool.py", line 83, in __init__ self.port = int(port) TypeError: int() argument must be a string or a number, not 'NoneType' Regards Santosh Hi, I am also facing similar issue. i download urllib3 and then try to run # python benchmark.py I got following exception .. i am using python 2.5 on Fedora 10 Running pool_get ... Traceback (most recent call last): File "benchmark.py", line 52, in <module> pool_get(TO_DOWNLOAD) File "benchmark.py", line 42, in pool_get pool = HTTPConnectionPool.from_url(url_list[0]) File "/usr/lib/python2.5/site-packages/urllib3-0.2-py2.5.egg/urllib3/connectionpool.py", line 120, in from_url return HTTPConnectionPool(host, port=port, timeout=timeout, maxsize=maxsize) File "/usr/lib/python2.5/site-packages/urllib3-0.2-py2.5.egg/urllib3/connectionpool.py", line 83, in __init__ self.port = int(port) TypeError: int() argument must be a string or a number, not 'NoneType' Regards Santosh Jan 17, 2009 issue 2 (Logging error if no Content-length header) reported by skilpat - What steps will reproduce the problem? 1. Download and run the benchmark. What is the expected output? What do you see instead? $ python urllib3/test/benchmark.py Running pool_get ... Traceback (most recent call last): File "urllib3/test/benchmark.py", line 52, in <module> pool_get(TO_DOWNLOAD) File "urllib3/test/benchmark.py", line 45, in pool_get r = pool.get_url(url) File "/Library/Python/2.5/site-packages/urllib3/connectionpool.py", line 247, in get_url return self.urlopen('GET', url, headers=headers, retries=retries, redirect=redirect) File "/Library/Python/2.5/site-packages/urllib3/connectionpool.py", line 213, in urlopen log.debug("\"%s %s %s\" %s %d" % (method, url, conn._http_vsn_str, httplib_response.status, httplib_response.length)) TypeError: int argument required (httplib_response.length is None, since content-length was not in the response) What version of the product are you using? On what operating system? urllib3 - r24 Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Please provide any additional information below. Also, the urllib3.HTTPResponse objects have no length attribute. -sk What steps will reproduce the problem? 1. Download and run the benchmark. What is the expected output? What do you see instead? $ python urllib3/test/benchmark.py Running pool_get ... Traceback (most recent call last): File "urllib3/test/benchmark.py", line 52, in <module> pool_get(TO_DOWNLOAD) File "urllib3/test/benchmark.py", line 45, in pool_get r = pool.get_url(url) File "/Library/Python/2.5/site-packages/urllib3/connectionpool.py", line 247, in get_url return self.urlopen('GET', url, headers=headers, retries=retries, redirect=redirect) File "/Library/Python/2.5/site-packages/urllib3/connectionpool.py", line 213, in urlopen log.debug("\"%s %s %s\" %s %d" % (method, url, conn._http_vsn_str, httplib_response.status, httplib_response.length)) TypeError: int argument required (httplib_response.length is None, since content-length was not in the response) What version of the product are you using? On what operating system? urllib3 - r24 Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Please provide any additional information below. Also, the urllib3.HTTPResponse objects have no length attribute. -sk Older Dec 02, 2008 issue 1 (Port is not optional (but should be)) reported by shazow - HTTPConnectionPool constructor casts port to an int which breaks the default port=None. This affects from_url, too. So, for 0.2, port is not optional. This will be fixed in future releases. HTTPConnectionPool constructor casts port to an int which breaks the default port=None. This affects from_url, too. So, for 0.2, port is not optional. This will be fixed in future releases. Nov 23, 2008 r24 (Added blocking support which kind of made the code a little ...) committed by shazow - Added blocking support which kind of made the code a little messier, but probably worth it. Trying to add stdout printing for unit tests, but not working yet. Added blocking support which kind of made the code a little messier, but probably worth it. Trying to add stdout printing for unit tests, but not working yet. Nov 23, 2008 r23 (Fixed up setup.py for pypi) committed by shazow - Fixed up setup.py for pypi Fixed up setup.py for pypi Nov 23, 2008 r22 (Tagged 0.2 release. ) committed by shazow - Tagged 0.2 release. Tagged 0.2 release. Nov 17, 2008 Examples (HTTPConnectionPool code examples) Wiki page edited by shazow

Examples (HTTPConnectionPool code examples) Wiki page commented on by rapsys_e...@yahoo.com - in my code i tried modifying the values of redirect (e.g. no value, true, false): {{{ # Make the request and capture the response try: request = self.vt_http.post_url('url_here', post_data, redirect=False) except urllib3.HTTPError, e: print "Error: " + e.strerror }}} also added this line inside connectionpool: {{{ print "URL1 %s\n" % response.headers.get('location') # Handle redirection if redirect and response.status in [301, 302, 303, 307] and 'location' in response.headers: # Redirect, retry print "URL2 %s\n" % response.headers.get('location') log.info("Redirecting %s -> %s" % (url, response.headers.get('location'))) return self.urlopen(method, response.headers.get('location'), body, headers, retries-1, redirect) return response }}} in my code if redirect is False, both URL1 and URL2 are not getting the correct URL if redirect is True, both URL1 and URL2 gets the correct URL but since redirect is true, another self.urlopen(...) will be called and follows the redirect im a newbie to this, but what i need is redirect=False, get url referrer and don't follow redirection... so that in my code, i can use the request.headers.get('location') and get the referrer... hope this makes sense...

in my code i tried modifying the values of redirect (e.g. no value, true, false): {{{ # Make the request and capture the response try: request = self.vt_http.post_url('url_here', post_data, redirect=False) except urllib3.HTTPError, e: print "Error: " + e.strerror }}} also added this line inside connectionpool: {{{ print "URL1 %s\n" % response.headers.get('location') # Handle redirection if redirect and response.status in [301, 302, 303, 307] and 'location' in response.headers: # Redirect, retry print "URL2 %s\n" % response.headers.get('location') log.info("Redirecting %s -> %s" % (url, response.headers.get('location'))) return self.urlopen(method, response.headers.get('location'), body, headers, retries-1, redirect) return response }}} in my code if redirect is False, both URL1 and URL2 are not getting the correct URL if redirect is True, both URL1 and URL2 gets the correct URL but since redirect is true, another self.urlopen(...) will be called and follows the redirect im a newbie to this, but what i need is redirect=False, get url referrer and don't follow redirection... so that in my code, i can use the request.headers.get('location') and get the referrer... hope this makes sense...

Today

Last 7 days

Last 30 days

Earlier this year

Older