The urlfetch API currently limits HTTP headers in both a production and SDK
environment. I would like the SDK to include values for these untrusted
request headers that mimic values used in production.

Some websites require a valid User-Agent string to process a request. The
presence of a User-Agent string identifies the request's originator for
tracking, permission, or other means.


Example:
Digg requires a User-Agent header for API requests. The site returns a 403
error if no User-Agent is present. App Engine SDK does not currently set
this header and I cannot override.
http://apidoc.digg.com/BasicConcepts#UserAgents


Proposed fix:
Add adjusted header values to urlfetch_stub.py to mimic a production
environment while differentiating production from localhost.

urlfetch_stub.py:
http://code.google.com/p/googleappengine/source/browse/trunk/google/appengine/api/urlfetch_stub.py

line 152:
"""
adjusted_headers = {
  'Content-Length': len(payload),
  'Host': host,
  'Accept': '*/*',
}
"""

new:
"""
adjusted_headers = {
  'Content-Length': len(payload),
  'Host': host,
  'Accept': '*/*',
  'User-Agent' : 'AppEngine-Google; (+http://code.google.com/appengine)
Development/1.0',
}
"""

	urlfetch_stub.py
	7.9 KB Download