My favorites | Sign in
Project Home
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 179: couchdb-dump cannot deal with unicode characters in doc ids
2 people starred this issue and may be notified of changes. Back to list
Status:  Fixed
Owner:  kxepal
Closed:  Apr 2013


Sign in to add a comment
 
Reported by heshim...@gmail.com, May 14, 2011
What steps will reproduce the problem?
1.Create a document in couchdb, with some Chinese character like "文档"
2.Run couchdb-dump on the database

What is the expected output? What do you see instead?
couchdb-dump crashes upon reaching this document. Here are the last lines of the trace:
  File "/pylonsenv/lib/python2.6/site-packages/couchdb/multipart.py", line 122, in __init__
    self._write_headers(headers)
  File "/pylonsenv/lib/python2.6/site-packages/couchdb/multipart.py", line 175, in _write_headers
    self.fileobj.write(headers[name])
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

What version of the product are you using? On what operating system?
couchdb-python 0.8 against couchdb 1.0.1 on Ubuntu.


May 14, 2011
#1 heshim...@gmail.com
I just needed a quick solution to dump the database and reload it in another environment. So I made some changes to multipart.py to get pass this utf-8 thing. It did work.

However, I understand that other parts are using multipart.py too. This probably won't fit the MIME standard. If I have time, I'll investigate further and provide a patch that does satisfy the MIME standard.
utf-8_dump_load.patch
1.1 KB   View   Download
May 14, 2011
#2 kxepal
Confirm. There is also invalid test case about how multipart module works with unicode data: StringIO could handle mixed "str" and "unicode" values, but files requires only "str" one.
May 14, 2011
#3 kxepal
Sorry, I was wrong about tests - StringIO confused me(: Don't rush, sit down and think about...yes(: 
There is no needs to fix multipart module, only dump tool due to it pass unicode document id to multipart writer. This is about dump-tool.patch.

dump-tool-2.patch solves same problem, but with respect of Content-Type header and his charset. I suppose, that would a more correct solution.
dump-tool.patch
648 bytes   View   Download
dump-tool-2.patch
1.5 KB   View   Download
May 14, 2011
#4 heshim...@gmail.com
Ah, that's much smarter. Thanks!
May 14, 2011
#5 heshim...@gmail.com
Hmm... another thing. I was under the impression that utf-8 encoded strings aren't valid ascii. Currently, isn't multipart.py expecting strict ascii strings as header?
May 14, 2011
#6 kxepal
Actually, only first 128 chars of utf-8 encoding are valid ascii. Problem was not in what characters in headers, but in type of string multipart tries to write into output stream. Files and streams doesn't expects pure unicode strings, but favors stings called as "bytes" in Python 3 terminology and multipart module expects this behavior.

But there was a "hack" which adds to headers document id which used by couchdb-load tool to help create document with same id value. Since document id could be unicode, this "hack" breaks expectations and makes multipart crash.

You could try revert patch and replace in dump.py default value of output argument in dump_db function from sys.stdout to StringIO.StringIO and error wouldn't be occurred because StringIO could handle both str and unicode values.
May 14, 2011
Project Member #7 djc.ochtman
IMO the correct way to have non-ASCII strings in MIME headers would be to use RFC 2047 encoding for any non-ascii header values.
May 14, 2011
#8 kxepal
Correct, but looks like an overhead in such case, because it would applied only to one header while others should follow RFC 822. Wouldn't be better to use base64 encoding?
Jun 1, 2011
#9 heshim...@gmail.com
Hmm... I'd like to make a note here that kxepal's dump-tool-2.patch actually generated some invalid multipart boundaries.
Sep 21, 2012
Project Member #10 djc.ochtman
(No comment was entered for this change.)
Owner: kxepal
Sep 21, 2012
#11 wickedg...@gmail.com
(No comment was entered for this change.)
Labels: Milestone-0.9
Oct 22, 2012
Project Member #12 djc.ochtman
Any progress on this?
Oct 22, 2012
#13 kxepal
Yes, will submit patch with tests during this week. I'd agreed with you about RFC 2047 specification, so diving into it.
Apr 24, 2013
#14 kxepal
Patch attached. Non-ascii headers now encoded following RFC 2047. Actually, I feel to rewrite multipart module to let him base on top of email package, but probably that would be another issue - need to workaround some email specific features to keep backward compatibility.
Status: Accepted
Apr 24, 2013
#15 kxepal
Sorry, forgot to cleanup testing prints. Reattached.
couchdb-python_485.patch
3.7 KB   View   Download
Apr 25, 2013
Project Member #16 djc.ochtman
Pushed a slightly changed patch as rce40fd77ae8d, thanks!
Status: Fixed
Apr 25, 2013
Project Member #17 djc.ochtman
(No comment was entered for this change.)
Labels: -Milestone-0.9
Sign in to add a comment

Powered by Google Project Hosting