Fixed
Status Update
No update yet.
Comments
p....@gmail.com
Forgot to mention that all this only happens in the cloud, dev server works fine.
p....@gmail.com
Ah, u'=C5=BCar=C3=B3wka1111111111111111'.decode('quotedprintable') breaks on my local
python 2.5 as well. 2.6 works fine, so nothing appengine specific here. It can be
worked around by doing .encode('ascii') on the unicode string first.
python 2.5 as well. 2.6 works fine, so nothing appengine specific here. It can be
worked around by doing .encode('ascii') on the unicode string first.
vi...@gmail.com
[Comment deleted]
sa...@gmail.com
This problem happens with additional parameters (to HTTP-POST), and it's very hard to work around because it encodes to base64 only if the string contains international characters.
ik...@google.com
ik...@google.com
Here's a workaround if the page is UTF-8:
# Upload page, starts blobstore session
class MainHandler(webapp.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload')
self.response.out.write('<html>')
self.response.out.write("""<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>""")
self.response.out.write('<body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" name="file"><br>
Some value: <input type="text" name="value" value="value"/>
<input type="submit" name="submit" value="Submit"> </form></body></html>""")
# Upload Handler, decodes and forces into correct format
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
upload_files = self.get_uploads('file') # 'file' is file upload field in the form
value = self.request.get("value")
value = unicode(base64.b64decode(value), "utf-8")
blob_info = upload_files[0]
self.redirect('/show/%s' % blob_info.key())
# Upload page, starts blobstore session
class MainHandler(webapp.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload')
self.response.out.write('<html>')
self.response.out.write("""<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>""")
self.response.out.write('<body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" name="file"><br>
Some value: <input type="text" name="value" value="value"/>
<input type="submit" name="submit" value="Submit"> </form></body></html>""")
# Upload Handler, decodes and forces into correct format
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
upload_files = self.get_uploads('file') # 'file' is file upload field in the form
value = self.request.get("value")
value = unicode(base64.b64decode(value), "utf-8")
blob_info = upload_files[0]
self.redirect('/show/%s' % blob_info.key())
vi...@gmail.com
This workaround results in a UnicodeError for me.
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 2: unexpected code byte
(The site is using utf-8)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 2: unexpected code byte
(The site is using utf-8)
mi...@gmail.com
Any updates??? The workaround is not working for me also. I am trying to post a form with text input and value 'Čučoriedka'...getting
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcb in position 1: unexpected end of data
BTW 'Čučoriedka' mingled into =C8u=E8oriedka
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcb in position 1: unexpected end of data
BTW 'Čučoriedka' mingled into =C8u=E8oriedka
vi...@gmail.com
Was disappointed that this was not fixed in last release. Basically this makes it impossible to include non-English text with a file upload. I would say that this is a pretty serious limitation of the Python runtime. It is more than a year since this bug was reported. Please fix it, or at least provide a monkey patch.
[Deleted User] <[Deleted User]>
The workaround that I have successfully implemented is to encode all values in base64 using JavaScript and decode them on the server. Far from ideal, but it got the job done.
p....@gmail.com
Hate to be spamming everyone, but could you please shed some light on these mysterious labels (squash-1 log-3482676)? They sound hopeful, but it would be nice to hear a definite answer. Should I be working around this problem or is it going to be fixed Really Soon Now?
tt...@google.com
I believe I've found a pretty simple solution to this. Please try adding accept-charset="utf-8" as an attribute to your blobstore <form> tag. This will force the user's browser to encode all string values in the form data as utf-8.
I've successfully tested this for a blobstore app of mine that I could break with unicode characters.
I've successfully tested this for a blobstore app of mine that I could break with unicode characters.
tt...@google.com
To be more clear, in a JSP it would look something like the following:
<form action="<%= blobstoreService.createUploadUrl("/upload") %>" method="post" enctype="multipart/form-data" accept-charset="utf-8">
and for Python it would look something like the following:
<form action="%s" method="POST" enctype="multipart/form-data" accept-charset="utf-8">' % upload_url
<form action="<%= blobstoreService.createUploadUrl("/upload") %>" method="post" enctype="multipart/form-data" accept-charset="utf-8">
and for Python it would look something like the following:
<form action="%s" method="POST" enctype="multipart/form-data" accept-charset="utf-8">' % upload_url
vi...@gmail.com
accept-charset will still return base64 encoded strings. Sometimes these string can't be decoded back to the user's input.
tm...@google.com
Sorry to have you waited so long.
Short version:
Try attached appengine_config.py if you're using webapp framework.
A bit longer version:
The root cause of this issue is that the webob module bundled in app engine SDK just ignores charset and transfer encoding setting specified in a header part of each multipart field. The attached file intends to override this behavior by monkey patching.
Could you try attached file and tell us how it goes?
The longest version:
I'm still drafting...
vi...@gmail.com
Thank you very much. I will try the monkey patch and report back.
pe...@google.com
br...@gmail.com
Is
ka...@gmail.com
I have this issue in latest SDK with russian letters :(
Solution from comment 21
Works for me
Solution from comment 21
Works for me
le...@gmail.com
I solved this problem by using chardet.
import chardet
def blob_decode(str):
encoding = chardet.detect(str)['encoding']
return str.decode(encoding)
text = blob_decode(base64.b64decode(text))
Using chardet, I could detect encoding of original text and decode properly.
import chardet
def blob_decode(str):
encoding = chardet.detect(str)['encoding']
return str.decode(encoding)
text = blob_decode(base64.b64decode(text))
Using chardet, I could detect encoding of original text and decode properly.
am...@frogsquared.com
I tried the appengine_config.py patch and it does appear to work except there appears to be a bug. The indentation for "obj.add(field.name , field_value)" causes field_value to be set from a prior loop when field.filename is present.
su...@gmail.com
Please Fix THIS!!!!
vi...@gmail.com
We have been using the patch with good results.
jf...@gmail.com
The patch is working for me. Will we have a permanent fix soon?
jm...@ies-sabadell.cat
THANK YOU very much. Patch works!!
an...@gmail.com
I'm hitting the same issue, but with the additional problem that JIS2004 Japanese characters are sometimes mis-encoded as ISO-2022-JP, meaning field_value.decode(field.type_options['charset']) doesn't work.
For example, the character ㈱ u'\u3351' does not exist in ISO-2022-JP, but when the blob-store mis-encodes it as '\033$B-H\033(B'. '\033$B-H\033(B'.decode('iso-2022-jp' fails.
So you need special case code of the form
if field.type_options['charset'].lower() == 'iso-2022-jp':
field_value = field_value.replace('\033$B', '\033$(Q').decode('iso-2022-jp-2004').encode(common_charset)
It seems reasonable to assume there will be special cases required for other encodings too, so this is hardly a satisfactory solution.
As it is, the choice of encoding seems completely random. For example a field containing the katakana テスト u'\u30c6\u30b9\u30c8' gets base64 encoded, whereas a field containing the katakana テキスト u'\u30c6\u30ad\u30b9\u30c8' is transcoded to charset=iso-2022-jp and sent 7bit.
Is it not possible to fix this properly so the blobstore leaves the content well alone, or at least chooses the encoding in a consistent manner that python regards as standards-compliant?
For example, the character ㈱ u'\u3351' does not exist in ISO-2022-JP, but when the blob-store mis-encodes it as '\033$B-H\033(B'. '\033$B-H\033(B'.decode('iso-2022-jp' fails.
So you need special case code of the form
if field.type_options['charset'].lower() == 'iso-2022-jp':
field_value = field_value.replace('\033$B', '\033$(Q').decode('iso-2022-jp-2004').encode(common_charset)
It seems reasonable to assume there will be special cases required for other encodings too, so this is hardly a satisfactory solution.
As it is, the choice of encoding seems completely random. For example a field containing the katakana テスト u'\u30c6\u30b9\u30c8' gets base64 encoded, whereas a field containing the katakana テキスト u'\u30c6\u30ad\u30b9\u30c8' is transcoded to charset=iso-2022-jp and sent 7bit.
Is it not possible to fix this properly so the blobstore leaves the content well alone, or at least chooses the encoding in a consistent manner that python regards as standards-compliant?
ab...@gmail.com
I'm using Django-nonrel and fixed this with a small middleware:
#------
import logging
import quopri
log = logging.getLogger(__name__)
class BlobRedirectFixMiddleware(object):
def process_request(self, request):
if request.method == 'POST' and 'HTTP_X_APPENGINE_BLOBUPLOAD' in request.META and request.META['HTTP_X_APPENGINE_BLOBUPLOAD'] == 'true':
request.POST = request.POST.copy()
log.info ('POST before decoding: %s' % request.POST)
for key in request.POST:
if key.startswith('_') or key == u'csrfmiddlewaretoken':
continue
value = request.POST[key]
if isinstance(value,(str, unicode)):
request.POST[key] = unicode(quopri.decodestring(value), 'iso_8859-2')
log.info ('POST after decoding: %s' % request.POST)
return None
#------
import logging
import quopri
log = logging.getLogger(__name__)
class BlobRedirectFixMiddleware(object):
def process_request(self, request):
if request.method == 'POST' and 'HTTP_X_APPENGINE_BLOBUPLOAD' in request.META and request.META['HTTP_X_APPENGINE_BLOBUPLOAD'] == 'true':
request.POST = request.POST.copy()
for key in request.POST:
if key.startswith('_') or key == u'csrfmiddlewaretoken':
continue
value = request.POST[key]
if isinstance(value,(str, unicode)):
request.POST[key] = unicode(quopri.decodestring(value), 'iso_8859-2')
return None
ni...@gmail.com
The python patch worked for me. Thanks a lot since I didn't understand what was going on when ÅÄÖ started displaying as xcTW.
he...@gmail.com
Any updates on this? The django nonrel solution above does not work, and it seems Content-Transfer-Encoding header is stripped when using Django-nonrel.
ma...@infium.se
I am writing an iOS application that will upload multilingual data to be stored in the datastore. Together with the data files are going to be stored in the blobstore. Getting correct character encoding is essential since many different languages will be used. One workaround would be to first create the data in datastore and then at a later stage add the files to blobstore. To avoid complexity I would prefer not to take that approach. Would upgrading to Python 2.7 do any good? Apart from this app engine is amazing !!
ni...@gmail.com
@mar...@infium.se: Using the patch supplied here works with python 2.7
http://code.google.com/p/googleappengine/issues/detail?id=2749#c21
You can try if adding that code works for you too.
You can try if adding that code works for you too.
ta...@gmail.com
Any instruction on using the patch above?
http://code.google.com/p/googleappengine/issues/detail?id=2749#c21
Clearly, adding the file to my root directory doesn't help.
I'm using python 2.7 with Django-nonrel.
Clearly, adding the file to my root directory doesn't help.
I'm using python 2.7 with Django-nonrel.
a....@gmail.com
this is still an issue. while it works on the local sdk it does not in production.
im using python 2.7 SDK 1.6.3
im using python 2.7 SDK 1.6.3
pa...@gmail.com
The middleware for change the post request in django, doesn't work because request.POST is inmutable. Although the content of your method works.
I created a method and I call it when I create the form, then if the form is valid or not the codification is right. My method return the new request.POST that is local.
crete_form = Form(decode(request))
Then when I use the values of the form I have tu use create_form.cleaned_data[key]
I created a method and I call it when I create the form, then if the form is valid or not the codification is right. My method return the new request.POST that is local.
crete_form = Form(decode(request))
Then when I use the values of the form I have tu use create_form.cleaned_data[key]
st...@gmail.com
The patch of Comment 21 worked for me, but it is strange that this bug is not yet fixed.
gu...@gmail.com
The patch in comment 21 also worked for me with Webapp2 and sdk release: "1.6.5" timestamp: 1332880663
ef...@gmail.com
[Comment deleted]
ch...@gmail.com
The patch in comment 21 fixed my error as well. Thank you for providing it.
ta...@gmail.com
The patch in comment 21 fixed my problem too with 2.7 environment.
SDK release: "1.6.6" timestamp: 1335306734
SDK release: "1.6.6" timestamp: 1335306734
na...@gmail.com
[Comment deleted]
na...@gmail.com
Just deploy the patch to the root of your app it will work even it might complain a little at local GAE launcher...
je...@gmail.com
The patch solved the blobstore encoding problem, but it will cause another error when using the image api document below:
https://developers.google.com/appengine/docs/python/images/usingimages
the error message:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\webapp\_webapp25.py", line 703, in __call__
handler.post (*groups)
File "F:\MyDoc\My Dropbox\workspace2\test25\src\test.py", line 22, in post
avatar = self.request.get("img")
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\webapp\_webapp25.py", line 161, in get
param_value = self.get_all(argument_name)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\webapp\_webapp25.py", line 195, in get_all
param_value = self.params.getall(argument_name)
File "C:\Program Files (x86)\Google\google_appengine\lib\webob_0_9\webob\__init__.py", line 951, in params
params = self.str_params
File "C:\Program Files (x86)\Google\google_appengine\lib\webob_0_9\webob\__init__.py", line 943, in str_params
return NestedMultiDict(self.str_GET, self.str_POST)
File "C:\Program Files (x86)\Google\google_appengine\lib\webob_0_9\webob\__init__.py", line 870, in str_POST
vars = MultiDict.from_fieldstorage(fs)
File "F:\MyDoc\My Dropbox\workspace2\test25\src\appengine_config.py", line 44, in from_fieldstorage
obj.add(field.name , field_value)
UnboundLocalError: local variable 'field_value' referenced before assignment
Please help.
the error message:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\webapp\_webapp25.py", line 703, in __call__
File "F:\MyDoc\My Dropbox\workspace2\test25\src\test.py", line 22, in post
avatar = self.request.get("img")
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\webapp\_webapp25.py", line 161, in get
param_value = self.get_all(argument_name)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\ext\webapp\_webapp25.py", line 195, in get_all
param_value = self.params.getall(argument_name)
File "C:\Program Files (x86)\Google\google_appengine\lib\webob_0_9\webob\__init__.py", line 951, in params
params = self.str_params
File "C:\Program Files (x86)\Google\google_appengine\lib\webob_0_9\webob\__init__.py", line 943, in str_params
return NestedMultiDict(self.str_GET, self.str_POST)
File "C:\Program Files (x86)\Google\google_appengine\lib\webob_0_9\webob\__init__.py", line 870, in str_POST
vars = MultiDict.from_fieldstorage(fs)
File "F:\MyDoc\My Dropbox\workspace2\test25\src\appengine_config.py", line 44, in from_fieldstorage
obj.add(
UnboundLocalError: local variable 'field_value' referenced before assignment
Please help.
ca...@gmail.com
[Comment deleted]
ca...@gmail.com
more than 2 year and it is still there.
Thank you google.
Thank you google.
ce...@gmail.com
Support is very slow :)
al...@cloudware.it
@comment 50: it's just an indentation issue on line 44 of the original patch, otherwise works fine.
Here's the same code with correct indentation.
Here's the same code with correct indentation.
tm...@google.com
Fixed in 1.7.2. Nou you can use webob-1.2.2 which doesn't have this issue.
pr...@google.com
You can require webob 1.2.2 for your application by adding the following to your app.yaml file (note this only work if you have runtime: python27):
libraries:
- name: webob
version: "1.2.2"
libraries:
- name: webob
version: "1.2.2"
a....@gmail.com
this still doesnt work!
if i upload a file with filename:
øoölssldkjff.jpg
i end up with:
=?ISO-8859-1?Q?=F8o=F6lssldkjff=2Ejpg?=
please reopen this issue.
if i upload a file with filename:
øoölssldkjff.jpg
i end up with:
=?ISO-8859-1?Q?=F8o=F6lssldkjff=2Ejpg?=
please reopen this issue.
ro...@gmail.com
To bypass the issue I use the upload instances to query the BlobInfo which do not have the issue
uploads = self.handler.get_uploads('...')
# here uploads might have shortened filenames which is a bug and makes them useless
blobs = BlobInfo.get([blob.key() for blob in uploads])
# here blobs have correct filenames
uploads = self.handler.get_uploads('...')
# here uploads might have shortened filenames which is a bug and makes them useless
blobs = BlobInfo.get([blob.key() for blob in uploads])
# here blobs have correct filenames
a....@gmail.com
thanks a lot but this is not a solution, its a workaround where i even have to get blobs again which is an overkill.
this has to be fixed and not worked around it.
this has to be fixed and not worked around it.
ro...@gmail.com
Yes I know this is just a bypass, but since I can't wait for the official fix, I use it.
And I think it's the only reliable way to have the complete filename : as I said filenames can be shortened in the upload instance so you can loose informations and for example file extension too (you can reproduce using long filename with several accents)
And I think it's the only reliable way to have the complete filename : as I said filenames can be shortened in the upload instance so you can loose informations and for example file extension too (you can reproduce using long filename with several accents)
pr...@google.com
Can you post a comment with your app.yaml file?
a....@gmail.com
here my app.yaml
pr...@google.com
Can you confirm that webob 1.2.2 is running by running:
>>> import webob
>>> print webob.__version__
>>> import webob
>>> print webob.__version__
a....@gmail.com
i setup a handler that prints the version out and it is 1.2.2 so yes it is the right version.
the remote api gives me:
s~xxxxxx> import webob
s~xxxxxx> webob.__version__
Traceback (most recent call last):
File "<console>", line 1, in <module>
AttributeError: 'module' object has no attribute '__version__'
the remote api gives me:
s~xxxxxx> import webob
s~xxxxxx> webob.__version__
Traceback (most recent call last):
File "<console>", line 1, in <module>
AttributeError: 'module' object has no attribute '__version__'
pr...@google.com
How are you printing the version if your handler?
a....@gmail.com
class WebObTest(BaseHandler):
def get(self):
import webob
version = webob.__version__
self.write(version)
pr...@google.com
I managed to reproduce this issue:
http://2749.proppy-bugs.appspot.com/
I believe this is another corner case that trigger a webob bug.
I believe this is another corner case that trigger a webob bug.
pr...@google.com
I checked that this is already fixed in webob HEAD with:
https://github.com/Pylons/webob/commit/df3fab28e348b5bdb51e5835d8833756acfc8399
This will be fixed on App Engine once webob release a new stable version including the fix.
This will be fixed on App Engine once webob release a new stable version including the fix.
[Deleted User] <[Deleted User]>
Webob 1.2.2 does not fix issue with random ecoding selection for all other fields of the form as what it does is just honor the Content-Transfer-Encoding if it was set for each part/field (and there is no way in HTML to force each FORM fields to declare it in MIME header).
The problem AFAIK, as it was pointed inhttp://stackoverflow.com/questions/3624226/encoding-problem-in-app-engine-when-submitting-multipart-form-data-forms )
The solution could be just let this handler leve content alone, not to decode/encode it as far as Content-Transfer-Encoding is not explicitly set.
Please, fix it as the world is not in English any more..neither it was in 2010 when the issue was opened.
The problem AFAIK, as it was pointed in
The solution could be just let this handler leve content alone, not to decode/encode it as far as Content-Transfer-Encoding is not explicitly set.
Please, fix it as the world is not in English any more..neither it was in 2010 when the issue was opened.
ce...@gmail.com
Employee some Japanese, Chineese, Polish, Russian people for unicode design/testing they will design and find unicode bugs easily what is very hard for English speaking people which not required unicode :)
ka...@gmail.com
Just to again confirm. This still happens, and it's still as moronic a bug as it was in Feb 2010.
ka...@gmail.com
A couple of other things:
1. The current stable release of Webob is now 1.2.3 which contains the commit referenced in #68
2. The latest SDK still only contains webob 1.1.1 and 0.9 - what's with that?
If that commit does indeed fix it, can we please get webob 1.2.3 in the 1.7.4 release?
1. The current stable release of Webob is now 1.2.3 which contains the commit referenced in #68
2. The latest SDK still only contains webob 1.1.1 and 0.9 - what's with that?
If that commit does indeed fix it, can we please get webob 1.2.3 in the 1.7.4 release?
a....@gmail.com
no webob 1.2.3 does not fix anything. but i also dont understand why the latest release does only have 1.1.1 available.
pr...@google.com
#73, can you share a test case that show the issue with webob 1.2.3.
When uploading øoölssldkjff.jpg it was correctly decoded in my request handler with 1.2.3.
When uploading øoölssldkjff.jpg it was correctly decoded in my request handler with 1.2.3.
a....@gmail.com
sorry my mistake... i meant 1.2.2, never tried 1.2.3 also because it was never available in any GAE version?!
pr...@google.com
I will try to make it available in the next release, in the meantime you can try to see if it fixes your issue by uploading a copy of webob 1.2.3 with your application.
bo...@gmail.com
Could you point me to the correct way of uploading a copy of webob 1.2.3 to my GAE project in python? Its no enough to just add the WebOb-1.2.3 folder to the root of the project right?
ma...@gmail.com
Just copying the /webob folder of the WebOb-1.2.3 distribution to the root of the project and uploading the application works for me. (In this case, no "libraries" entry for webob is needed in app.yaml.)
ju...@gmail.com
Thank you Romain (
I would of course still prefer a proper fix so that blob_info.filename works!
I would of course still prefer a proper fix so that blob_info.filename works!
ab...@gmail.com
Trying to use 1.2.2 I get the following error :
Error parsing yaml file:
webob version "1.2.2" is not supported, use one of: "1.1.1" or "latest" ("latest" recommended for development only)
Thanks.
Error parsing yaml file:
webob version "1.2.2" is not supported, use one of: "1.1.1" or "latest" ("latest" recommended for development only)
Thanks.
pr...@google.com
This will be corrected in the next release, with webob 1.2.3, in the meantime please use the workaround pointed in #78
ab...@gmail.com
This fixed the first problem. However the fields that are non-ascii are coming down as encoded (utf-8 in my case). Which is unexpected. I am working around by attempting to encode just to check and decode if encoding fails...This fixed the first problem. However the fields that are non-ascii are coming down as encoded (utf-8 in my case). Which is unexpected. I am working around by attempting to encode just to check and decode if encoding fails...
ab...@gmail.com
How bad your monkey patch suggestion is !! This broke all form submits now. I have not looked into the details for the problem, but I suspect that all fields are now coming down as un-decoded !! (Bad idea to suggest this without testing on your side. And no response to my comment earlier about decode problems, were my suspicion is true. I had mistakenly assumed that this was only breaking multi-form submits, but it is not).
Please let me know if I am doing anything wrong.
Please let me know if I am doing anything wrong.
ab...@gmail.com
Have you seen #78 and tried that yet?
ab...@gmail.com
Sorry I cannot experiment any more.. What you are suggesting is a big change and it is not officially out there as a production version by Google App Engine.. When are you putting it out as a production version.
Also you have an explanation for the decoding issue ? What is causing for non-multi-part forms to receive fields that are not decoded ?
Also you have an explanation for the decoding issue ? What is causing for non-multi-part forms to receive fields that are not decoded ?
a....@gmail.com
ok so tested this again. by referencing kebob 1.2.3 in app.yaml and even by uploading this version with my app and i still end up with weird encodings.
when working with the SDK it works fine even with webob 1.1 ending up with the right filename.
in production it just does not work whatever the webob version is (and yes webob 1.2.3 is loaded in production).
uploading a file with filename øoölssldkjff.jpg ends up in =?ISO-8859-1?Q?=F8o=F6lssldkjff=2Ejpg?=
when working with the SDK it works fine even with webob 1.1 ending up with the right filename.
in production it just does not work whatever the webob version is (and yes webob 1.2.3 is loaded in production).
uploading a file with filename øoölssldkjff.jpg ends up in =?ISO-8859-1?Q?=F8o=F6lssldkjff=2Ejpg?=
pr...@google.com
This bug was about additional form fields encoding getting broken when being part of a file upload POST request.
I confirm that it works for me with webob 1.2.3 (1.7.3), see:
http://2749.proppy-bugs.appspot.com/
Are you seeing a different behavior, can you supply a sample code to reproduce the issue?
I confirm that it works for me with webob 1.2.3 (1.7.3), see:
Are you seeing a different behavior, can you supply a sample code to reproduce the issue?
[Deleted User] <[Deleted User]>
After adding webob 1.2.3 in my yaml its working now. In my case I was uploading a audio file along with title(in bangla language) using multipart request. In server end my title was messed up before and now it fixed.
Before:
dGl0bGU9w5bDpuKBhMOuw7vLnsuZIMufw7nigYTDn+KBhMO+IOKAusO84oKnw67igJrDn8ufw7nL
h8ugw7Ygw6HDrcufw7nigYTLmV8sX3VuaWNvZGVfdGl0bGU94KaH4Kac4Kak4KeH4Kau4Ka+4Kav
4Ka8IOCmrOCmv+CmleCnh+CmsuCnhyDgpq/gp4zgpqTgp4HgppXgpqzgpr/gprngp4Dgpqgg4KaX
4Kaj4Kas4Ka/4Kav4Ka84KeHXyxfbmV3c19zb3VyY2U9UHJvdGhvbS1hbG9fLF9pc191bmljb2Rl
PXRydWVfLF9pc19iYW5nbGE9dHJ1ZQ==
After:
unicode_title=যুদ্ধেও নিষিদ্ধ পিপার স্প্রে, কিন্তু স্বরাষ্ট্রমন্ত্রীর নির্দেশে ব্যবহার_,_news_source=UK BD News_,_is_unicode=true_,_is_bangla=true
Before:
dGl0bGU9w5bDpuKBhMOuw7vLnsuZIMufw7nigYTDn+KBhMO+IOKAusO84oKnw67igJrDn8ufw7nL
h8ugw7Ygw6HDrcufw7nigYTLmV8sX3VuaWNvZGVfdGl0bGU94KaH4Kac4Kak4KeH4Kau4Ka+4Kav
4Ka8IOCmrOCmv+CmleCnh+CmsuCnhyDgpq/gp4zgpqTgp4HgppXgpqzgpr/gprngp4Dgpqgg4KaX
4Kaj4Kas4Ka/4Kav4Ka84KeHXyxfbmV3c19zb3VyY2U9UHJvdGhvbS1hbG9fLF9pc191bmljb2Rl
PXRydWVfLF9pc19iYW5nbGE9dHJ1ZQ==
After:
unicode_title=যুদ্ধেও নিষিদ্ধ পিপার স্প্রে, কিন্তু স্বরাষ্ট্রমন্ত্রীর নির্দেশে ব্যবহার_,_news_source=UK BD News_,_is_unicode=true_,_is_bangla=true
mr...@gmail.com
As mentioned in #79 it seems like blob_info.filename is still doing this in production (observed on python using 1.7.4 with webob 1.2.3 included)
Does that require a new issue or should this be reopened?
Does that require a new issue or should this be reopened?
lu...@gmail.com
[Comment deleted]
[Deleted User] <[Deleted User]>
This bug still happens with webob 1.2.3
The problem is that additional form fields that accompany a file upload have \r\n= every 75 characters.
The problem is that additional form fields that accompany a file upload have \r\n= every 75 characters.
ey...@gmail.com
Still getting this error as stated above. I'm uploading a text file with some JSON metadata, and the JSON has \r\n= every 75 characters....very annoying.
{"platformVersion":"16","platform":"ANDROID","logContent":"abcdefghijklmnop=
qrstuvw","appVersion":"1.0.17","userMessage":"Test Test Test Test Test Tes =
t Test","device":"Motorola DROID RAZR HD","uuid":"2c4d836b-7ea8-3494-91e0-3=
2b3f2f07e00","appType":"ra","logType":"SETTINGS"}
{"platformVersion":"16","platform":"ANDROID","logContent":"abcdefghijklmnop=
qrstuvw","appVersion":"1.0.17","userMessage":"Test Test Test Test Test Tes =
t Test","device":"Motorola DROID RAZR HD","uuid":"2c4d836b-7ea8-3494-91e0-3=
2b3f2f07e00","appType":"ra","logType":"SETTINGS"}
ey...@gmail.com
Also, when I send the same request to a webapp2.RequestHandler everything is fine.
pr...@google.com
Sorry for the late followup, can you confirm that your application is using webob 1.2.3 ?
`print webob.__version__` inside your application.
If that's not the case you can force it by adding the following lines to your app.yaml:
libraries:
- name: webob
version: "1.2.3"
`print webob.__version__` inside your application.
If that's not the case you can force it by adding the following lines to your app.yaml:
libraries:
- name: webob
version: "1.2.3"
ey...@gmail.com
Please disregard previous comment. I thought my production app.yaml forced 1.2.3, but that was only in my test one. Sorry about that.
ki...@gmail.com
This should be mentioned in the doc (i.e. at https://developers.google.com/appengine/docs/python/tools/webapp/blobstorehandlers )
What bugs me is that there are discrepancies between the dev server and production if you don't specify your webob version.
What bugs me is that there are discrepancies between the dev server and production if you don't specify your webob version.
li...@gmail.com
Hello everyone I'm very new to python and google app engine, I also have this issue when I'm using django + GAE...
It's getting me so confused because a single post data like "google is a custom workshop experience designed to help customers grow their capabilities. We partner with brands to advance their digital strategies by exploring how to connect with consumers across a diverse series of segments and maximize Google’s tools and technologies to bring
their stories to life online." gets break and strangely add "=\r\n" inside the string.
I don't know how to use webob with django, and I've found so few information about this issue. So I'm just using a very stupid solution that I "transfer" all the request.POST data before I use them.
def postDataHandler(request):
for key in request.POST:
if key.startswith('_') or key == u'csrfmiddlewaretoken':
continue
value = request.POST[key]
if isinstance(value,(str, unicode)):
request.POST[key] = unicode(quopri.decodestring(value), 'iso_8859-2')
return request
It's very ugly but working.
Anyway it will be so helpful if anyone can provide me a better solution for this
It's getting me so confused because a single post data like "google is a custom workshop experience designed to help customers grow their capabilities. We partner with brands to advance their digital strategies by exploring how to connect with consumers across a diverse series of segments and maximize Google’s tools and technologies to bring
their stories to life online." gets break and strangely add "=\r\n" inside the string.
I don't know how to use webob with django, and I've found so few information about this issue. So I'm just using a very stupid solution that I "transfer" all the request.POST data before I use them.
def postDataHandler(request):
for key in request.POST:
if key.startswith('_') or key == u'csrfmiddlewaretoken':
continue
value = request.POST[key]
if isinstance(value,(str, unicode)):
request.POST[key] = unicode(quopri.decodestring(value), 'iso_8859-2')
return request
It's very ugly but working.
Anyway it will be so helpful if anyone can provide me a better solution for this
dj...@gmail.com
It is still broken.
Why do encoding a filename of BlobInfo with MIME?
We do not want. Please return a unicode value encoded with UTF-8.
For all. You can get a correct filename, if you using any character code, as below:
import email
for blob_info in self.get_uploads('file'):
filename_mime = blob_info.filename
if isinstance(filename_mime, unicode):
filename_mime_utf8 = filename_mime.encode('utf-8')
else:
filename_mime_utf8 = filename_mime
filename_encoded, encoding = email.header.decode_header(filename_mime_utf8)[0]
if encoding is not None:
filename_unicode = filename_encoded.decode(encoding)
filename_utf8 = filename_unicode.encode('utf-8')
blob_info._BlobInfo__entity['filename'] = filename_utf8
Why do encoding a filename of BlobInfo with MIME?
We do not want. Please return a unicode value encoded with UTF-8.
For all. You can get a correct filename, if you using any character code, as below:
import email
for blob_info in self.get_uploads('file'):
filename_mime = blob_info.filename
if isinstance(filename_mime, unicode):
filename_mime_utf8 = filename_mime.encode('utf-8')
else:
filename_mime_utf8 = filename_mime
filename_encoded, encoding = email.header.decode_header(filename_mime_utf8)[0]
if encoding is not None:
filename_unicode = filename_encoded.decode(encoding)
filename_utf8 = filename_unicode.encode('utf-8')
blob_info._BlobInfo__entity['filename'] = filename_utf8
tj...@gmail.com
I can confirm that adding
libraries:
- name: webob
version: "1.2.3"
to app.yaml solved the problem for me.
libraries:
- name: webob
version: "1.2.3"
to app.yaml solved the problem for me.
ar...@gmail.com
#100 did not work for me. #99 did.
cm...@gmail.com
The problem is still there for Java SDK. I'm trying to upload a file with thai filename. It seems like the filename encoding ends up something like "=?UTF-8?B?4Lit4Liw4LmE4LijLlBORw==?=" from "อะไร.PNG".
I've tried everything to fix this problem. It seems like on my development server. The filename is encoded as ISO8859_1 which i've managed to change the encoding to utf8 correctly. But on the deployment server filename is completely ruined.
Please help!
I've tried everything to fix this problem. It seems like on my development server. The filename is encoded as ISO8859_1 which i've managed to change the encoding to utf8 correctly. But on the deployment server filename is completely ruined.
Please help!
gu...@gmail.com
I confirm this is broken in the Java production environment - getting =?UTF-8?... weird string as filename. I remember successfully getting non-english characters in filenames in the past, but today I found out it doesn't work, so it should be a relatively recent bug. I tried setting the accept-charset to 'utf-8' but it didn't help.
dc...@gmail.com
Still broken, and in Go too.
This is a 5 years old bug, Google!
This is a 5 years old bug, Google!
ch...@gmail.com
Hi guys,
We resolved to do another thing. Instead of appengine_config.py and app.yaml modifications we used a two-step handlers approach using poster.encode module.
HANDLER 1 (webapp2.RequestHandler):
(1) receives a form with action="post" and enctype="multipart/form-data" including the file that will go to the blobstore and other elements we wish to have in our datastore
(2) then, it generates an entity and takes the generated entity id to make a urlfetch with a "multipart/form-data"-poster.encode generated file, to post to the HANDLER 2
This is my handler for a blog, take a look at line where upload_url is generated, this a key moment when I resend my entity id to the blobstore handler:
def post(self, post_id):
if post_id == '1':
blog = models.BlogPost()
blog.title = self.request.get('title')
blog.subtitle = self.request.get('subtitle')
blog.author = self.request.get('author')
blog.brief = self.request.get('brief')
blog.content = self.request.get('content')
blog.category = self.request.get('category').split(',')
# blog.blob_key is allocated in second handler
blog.put()
else:
blog = models.BlogPost.get_by_id(long(post_id))
if blog is not None:
blog.title = self.request.get('title')
blog.subtitle = self.request.get('subtitle')
blog.author = self.request.get('author')
blog.brief = self.request.get('brief')
blog.content = self.request.get('content')
blog.category = self.request.get('category').split(',')
# blog.blob_key is allocated in second handler
blog.put()
#re-post to blobstore to allocate blog.blob_key as a BlobKeyProperty
from google.appengine.api import urlfetch
from poster.encode import multipart_encode, MultipartParam
payload = {}
file_data =self.request.POST ['file']
payload['file'] = MultipartParam('file', filename=file_data.filename,
filetype=file_data.type,
fileobj=file_data.file)
data,headers= multipart_encode(payload)
####### UPLOAD URL #######
upload_url = blobstore.create_upload_url('/admin/blog/upload/%s/' %blog.key.id ())
t = urlfetch.fetch(url=upload_url, payload="".join(data), method=urlfetch.POST, headers=headers)
#output toast message
if t.content == 'success':
self.add_message('Changes saved', 'success')
return self.get(post_id =blog.key.id ())
else:
self.add_message('Something bad happened', 'danger')
return self.get(post_id =blog.key.id ())
HANDLER 2 (blobstore_handlers.BlobstoreUploadHandler):
def post(self, post_id):
try:
blog = models.BlogPost.get_by_id(long(post_id))
try:
blobstore.delete(blog.blob_key) #just in case we had a previous blob_key allocated, we get rid of the old one
except:
pass
upload = self.get_uploads()[0]
blog.blob_key = upload.key()
blog.put()
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('success')
except Exception as e:
logging.error('something went wrong: %s' % e)
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('error')
Voilá, you got all your data without being buggly affected by the blobstore handler. Additionally, this way you also avoid expiration from blobstore.create_upload_url so you can take your time to submit any long post.
---
BlogPost Model:https://dpaste.de/RPN0
We resolved to do another thing. Instead of appengine_config.py and app.yaml modifications we used a two-step handlers approach using poster.encode module.
HANDLER 1 (webapp2.RequestHandler):
(1) receives a form with action="post" and enctype="multipart/form-data" including the file that will go to the blobstore and other elements we wish to have in our datastore
(2) then, it generates an entity and takes the generated entity id to make a urlfetch with a "multipart/form-data"-poster.encode generated file, to post to the HANDLER 2
This is my handler for a blog, take a look at line where upload_url is generated, this a key moment when I resend my entity id to the blobstore handler:
def post(self, post_id):
if post_id == '1':
blog = models.BlogPost()
blog.title = self.request.get('title')
blog.subtitle = self.request.get('subtitle')
blog.author = self.request.get('author')
blog.brief = self.request.get('brief')
blog.content = self.request.get('content')
blog.category = self.request.get('category').split(',')
# blog.blob_key is allocated in second handler
blog.put()
else:
blog = models.BlogPost.get_by_id(long(post_id))
if blog is not None:
blog.title = self.request.get('title')
blog.subtitle = self.request.get('subtitle')
blog.author = self.request.get('author')
blog.brief = self.request.get('brief')
blog.content = self.request.get('content')
blog.category = self.request.get('category').split(',')
# blog.blob_key is allocated in second handler
blog.put()
#re-post to blobstore to allocate blog.blob_key as a BlobKeyProperty
from google.appengine.api import urlfetch
from poster.encode import multipart_encode, MultipartParam
payload = {}
file_data =
payload['file'] = MultipartParam('file', filename=file_data.filename,
filetype=file_data.type,
fileobj=file_data.file)
data,headers= multipart_encode(payload)
####### UPLOAD URL #######
upload_url = blobstore.create_upload_url('/admin/blog/upload/%s/' %
t = urlfetch.fetch(url=upload_url, payload="".join(data), method=urlfetch.POST, headers=headers)
#output toast message
if t.content == 'success':
self.add_message('Changes saved', 'success')
return self.get(post_id =
else:
self.add_message('Something bad happened', 'danger')
return self.get(post_id =
HANDLER 2 (blobstore_handlers.BlobstoreUploadHandler):
def post(self, post_id):
try:
blog = models.BlogPost.get_by_id(long(post_id))
try:
blobstore.delete(blog.blob_key) #just in case we had a previous blob_key allocated, we get rid of the old one
except:
pass
upload = self.get_uploads()[0]
blog.blob_key = upload.key()
blog.put()
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('success')
except Exception as e:
logging.error('something went wrong: %s' % e)
self.response.headers['Content-Type'] = 'text/plain'
self.response.out.write('error')
Voilá, you got all your data without being buggly affected by the blobstore handler. Additionally, this way you also avoid expiration from blobstore.create_upload_url so you can take your time to submit any long post.
---
BlogPost Model:
Description
class DebugHandler(blobstore_handlers.BlobstoreUploadHandler):
def get(self):
self.response.out.write(blobstore.create_upload_url('/blobs/debug'))
def post(self):
filename = self.request.get("filename")
logging.debug(repr(filename))
test:
$ url=..../blobs/debug
$ curl -F "filename=żarówka" $url $(curl $url) # żarówka is a polish word
with some unicode chars
Notice that the data is submitted to both $url and the upload_url which we
get from the subshelled GET. Results:
DEBUG: u'\u017car\xf3wka' - correct after direct submission to $url
DEBUG: u'xbxhcsOzd2th' - ouch, base64 encoded after the blobstore mangling
Even more fun:
$ curl -F "filename=żarówka1111111111111111" $url $(curl $url)
DEBUG: u'\u017car\xf3wka1111111111111111'
DEBUG: u'=C5=BCar=C3=B3wka1111111111111111' - quotedprintable encoded this
time...
When trying to work around this bug I stumbled upon another one, check on
>>> u'=C5=BCar=C3=B3wka1111111111111111'.decode('quotedprintable')
'=\x00\x00\x00C\x00\x00\x005\x00\x00\x00=\x00\x00\x00B\x00\x00\x00C\x00\x00
\x00a\x00\x00\x00r\x00\x00\x00=\x00\x00\x00C\x00\x00\x003\x00\x00\x00=\x00\
x00\x00B\x00\x00\x003\x00\x00\x00w\x00\x00\x00k\x00\x00\x00a\x00\x00\x001\x
00\x00\x001\x00\x00\x001\x00\x00\x001\x00\x00\x001\x00\x00\x001\x00\x00\x00
1\x00\x00\x001\x00\x00\x001\x00\x00\x001\x00\x00\x001\x00\x00\x001\x00\x00\
x001\x00\x00\x001\x00\x00\x001\x00\x00\x001\x00\x00\x00'
>>> '=C5=BCar=C3=B3wka1111111111111111'.decode('quotedprintable')
'\xc5\xbcar\xc3\xb3wka1111111111111111'