My favorites | Sign in
Project Home
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 147: support Google App Engine's URL fetch service
1 person starred this issue and may be notified of changes. Back to list
Status:  New
Owner:  ----


Sign in to add a comment
 
Reported by acmo...@gmail.com, Aug 30, 2010
What steps will reproduce the problem?
1. use Google's App Engine SDK
2. install couchdb-python into your new project
3. import couchdb
2. attempt to connect to an external couchdb server with couch = couchdb.Server('https://couchdb.example.com/')
3. exception is raised

What is the expected output? What do you see instead?

Google has their own url fetch service in their app engine. <https://code.google.com/appengine/docs/python/urlfetch/overview.html>. Other than that, they appear to prevent applications from making external socket connections. It appears to me that since this code uses couchdb/http.py, it won't work on Google's app engine. Here's the stacktrace:

Traceback (most recent call last):
  File "/home/amoore/dev/google_appengine/google/appengine/ext/webapp/__init__.py", line 511, in __call__
    handler.get(*groups)
  File "/home/amoore/dev/emradmin/main.py", line 13, in get
    userdb = couch['user']
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 137, in __getitem__
    db.resource.head() # actually make a request to the database
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 377, in head
    return self._request('HEAD', path, headers=headers, **params)
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 419, in _request
    credentials=self.credentials)
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 239, in request
    resp = _try_request_with_retries(iter(self.retry_delays))
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 196, in _try_request_with_retries
    return _try_request()
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 211, in _try_request
    if conn.sock is None:
AttributeError: HTTPConnection instance has no attribute 'sock'

If urllib, urllib2 or httplib were used instead, I think this would work.

What version of the product are you using? On what operating system?
python 2.5.2, Google App Engine, linux


Aug 30, 2010
#1 matt.goo...@gmail.com
couchdb.http is actually just a layer over httplib so, in theory, it should work. Unfortunately, couchdb.http uses an undocumented 'sock' attribute of httplib.HTTPConnection that the Google API doesn't seem to provide.

'sock' is primarily used to re-open dead, cached connections but I don't see any other obvious, i.e. documented, way to test if a connection is open. I guess we could just rely on the retry code picking it up.

Note: couchdb.http also uses conn.sock.sendall but I suspect that could easily be replaced with conn.send.
Labels: -Type-Defect Type-Enhancement
Aug 30, 2010
#2 acmo...@gmail.com
Aha. That makes sense. If you come up with a potential fix, and you don't want to jump through the hoops of getting an App Engine account, I can try it out for you. In the meantime, I'll try to hack my way into a "fix".

Thanks!
Aug 31, 2010
#3 acmo...@gmail.com
Thanks for the pointers. I've attached a patch that removes the calls to 'sock'. The test suite seems to do about the same (there are 3 failures before and after the patch), and this patch works in Google's App Engine. I don't really know how to test the effects on timeouts.
couchdb_google.diff
1.7 KB   View   Download
Aug 31, 2010
#4 acmo...@gmail.com
hrm. It looks like they don't implement a 'isclosed' method, either:

  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 292, in __iter__
    return iter([item.id for item in self.view('_all_docs')])
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 984, in __iter__
    return iter(self.rows)
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 1003, in rows
    self._fetch()
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 990, in _fetch
    data = self.view._exec(self.options)
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/client.py", line 880, in _exec
    _, _, data = self.resource.get_json(**self._encode_options(options))
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 394, in get_json
    data = json.decode(data.read())
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 96, in read
    self.close()
  File "/home/amoore/dev/emradmin/lib/python2.5/site-packages/couchdb/http.py", line 100, in close
    while not self.resp.isclosed():
  File "/home/amoore/dev/google_appengine/google/appengine/dist/httplib.py", line 231, in __getattr__
    return getattr(self.fp, attr)
AttributeError: StringIO instance has no attribute 'isclosed'

This means you can't iterate over a database.

I'll see if I can work around this, too.


Sep 12, 2010
#5 polestn...@gmail.com
So, what about GAE support?
Sep 19, 2010
Project Member #6 djc.ochtman
acmoore: that patch looks solid. Can you give me your name and an email address so we can credit you on the changeset?
Sep 19, 2010
#7 amo...@mooresystems.com
You can credit it to: Andrew Moore <amoore@mooresystems.com>. Thank you!
I'm also willing to assign copyright to one of you, in case that helps with future maintenance.

I have a patch for the 'isclosed' problem, too. It's necessary in order to iterate over databases inside the GAE. I'll find it, clean it up, and submit it, too.

Sep 19, 2010
Project Member #8 djc.ochtman
Pushed as r25df81c5d704. Would be great if you come up with the other patch! Can you also show us what kind of test suite failures we get on GAE? Also, apparently we don't have iterating over databases covered in the test suite? Would be nice to fix that, too.
Sep 19, 2010
#9 amo...@mooresystems.com
Thanks for pushing out that previous patch.

Here's the patch I'm using to get around the problem with .isclosed(). Now that I look at it, I'm not sure that it is only working because I always happen to get all of the data in the first read or not. Works for me, though.

isclosed.diff
647 bytes   View   Download
Jul 15, 2014
Project Member #10 djc.ochtman
This issue has been migrated to GitHub. Please continue discussion here:

https://github.com/djc/couchdb-python/issues/147
Sign in to add a comment

Powered by Google Project Hosting