mem_issue.txt - georgeyk - Project Hosting on Google Code

r70

Source path: svn/ trunk/ gsoc2008/ mem_issue.txt

***
Here is just my thoughts about memory leak issue in the libgettextpo python
interface in the translate toolkit (storage/cpo.py)
***

***
My comments with a '->>' mark (hint: grep -C N)
***


->> Running some tests

$ py.test -k plurals
===================================================================== test process starts =====================================================================
executable:   /usr/bin/python  (2.5.2-final-0)
using py lib: /usr/lib/python2.5/site-packages/py <rev unknown>

test_base.py[17] sssssssssssssssss
test_cpo.py[58] sssssss.ssssssssssssssssssssssssssssss*** glibc detected *** /usr/bin/python: munmap_chunk(): invalid pointer: 0x00007fa22e510514 ***
======= Backtrace: =========
/lib/libc.so.6(cfree+0x1b6)[0x7fa230f3ad46]
/usr/lib/libgettextpo.so.0(po_message_iterator_free+0xd)[0x7fa22e2a7d3d]
/usr/lib/python2.5/lib-dynload/_ctypes.so(ffi_call_unix64+0x4c)[0x7fa22eb3faec]
/usr/lib/python2.5/lib-dynload/_ctypes.so(ffi_call+0x285)[0x7fa22eb3f975]
/usr/lib/python2.5/lib-dynload/_ctypes.so(_CallProc+0x2b7)[0x7fa22eb3a907]
/usr/lib/python2.5/lib-dynload/_ctypes.so[0x7fa22eb3455f]
/usr/bin/python(PyObject_Call+0x13)[0x417e33]
/usr/bin/python(PyEval_EvalFrameEx+0x314a)[0x48617a]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalCodeEx+0x776)[0x48a406]
/usr/bin/python[0x4d528a]
/usr/bin/python(PyObject_Call+0x13)[0x417e33]
/usr/bin/python[0x41e66f]
/usr/bin/python(PyObject_Call+0x13)[0x417e33]
/usr/bin/python(PyEval_CallObjectWithKeywords+0x72)[0x482052]
/usr/bin/python[0x4632a0]
/usr/bin/python[0x459928]
/usr/bin/python(PyEval_EvalFrameEx+0x8d3)[0x483903]
/usr/bin/python(PyEval_EvalCodeEx+0x776)[0x48a406]
/usr/bin/python[0x4d528a]
/usr/bin/python(PyObject_Call+0x13)[0x417e33]
/usr/bin/python(PyEval_EvalFrameEx+0x4043)[0x487073]
/usr/bin/python(PyEval_EvalCodeEx+0x776)[0x48a406]
/usr/bin/python[0x4d528a]
/usr/bin/python(PyObject_Call+0x13)[0x417e33]
/usr/bin/python(PyEval_EvalFrameEx+0x4043)[0x487073]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalFrameEx+0x5bd7)[0x488c07]
/usr/bin/python(PyEval_EvalCodeEx+0x776)[0x48a406]
/usr/bin/python(PyEval_EvalFrameEx+0x5045)[0x488075]
/usr/bin/python(PyEval_EvalCodeEx+0x776)[0x48a406]
/usr/bin/python(PyEval_EvalCode+0x32)[0x48a522]
/usr/bin/python(PyRun_FileExFlags+0x10e)[0x4abe2e]
/usr/bin/python(PyRun_SimpleFileExFlags+0x1a9)[0x4ac0c9]
/usr/bin/python(Py_Main+0x8fd)[0x4145ad]
/lib/libc.so.6(__libc_start_main+0xf4)[0x7fa230ee11c4]
/usr/bin/python[0x413b29]
======= Memory map: ========
00400000-00522000 r-xp 00000000 08:04 1786559                            /usr/bin/python2.5
00721000-00753000 rw-p 00121000 08:04 1786559                            /usr/bin/python2.5
00753000-00a3e000 rw-p 00753000 00:00 0                                  [heap]
7fa22e08f000-7fa22e09c000 r-xp 00000000 08:04 1171519                    /lib/libgcc_s.so.1
7fa22e09c000-7fa22e29c000 ---p 0000d000 08:04 1171519                    /lib/libgcc_s.so.1
7fa22e29c000-7fa22e29d000 rw-p 0000d000 08:04 1171519                    /lib/libgcc_s.so.1
7fa22e29d000-7fa22e2dd000 r-xp 00000000 08:04 1788683                    /usr/lib/libgettextpo.so.0.4.0
7fa22e2dd000-7fa22e4dc000 ---p 00040000 08:04 1788683                    /usr/lib/libgettextpo.so.0.4.0
7fa22e4dc000-7fa22e4df000 rw-p 0003f000 08:04 1788683                    /usr/lib/libgettextpo.so.0.4.0
7fa22e4df000-7fa22e520000 rw-p 7fa22e4df000 00:00 0 
7fa22e520000-7fa22e523000 r-xp 00000000 08:04 1860403                    /usr/lib/python2.5/lib-dynload/_random.so
7fa22e523000-7fa22e722000 ---p 00003000 08:04 1860403                    /usr/lib/python2.5/lib-dynload/_random.so
7fa22e722000-7fa22e723000 rw-p 00002000 08:04 1860403                    /usr/lib/python2.5/lib-dynload/_random.so
7fa22e723000-7fa22e728000 r-xp 00000000 08:04 1860413                    /usr/lib/python2.5/lib-dynload/binascii.so
7fa22e728000-7fa22e927000 ---p 00005000 08:04 1860413                    /usr/lib/python2.5/lib-dynload/binascii.so
7fa22e927000-7fa22e928000 rw-p 00004000 08:04 1860413                    /usr/lib/python2.5/lib-dynload/binascii.so
7fa22e928000-7fa22e92b000 r-xp 00000000 08:04 1860421                    /usr/lib/python2.5/lib-dynload/math.so
7fa22e92b000-7fa22eb2b000 ---p 00003000 08:04 1860421                    /usr/lib/python2.5/lib-dynload/math.so
7fa22eb2b000-7fa22eb2c000 rw-p 00003000 08:04 1860421                    /usr/lib/python2.5/lib-dynload/math.so
7fa22eb2c000-7fa22eb44000 r-xp 00000000 08:04 1861222                    /usr/lib/python2.5/lib-dynload/_ctypes.so
7fa22eb44000-7fa22ed44000 ---p 00018000 08:04 1861222                    /usr/lib/python2.5/lib-dynload/_ctypes.so
7fa22ed44000-7fa22ed48000 rw-p 00018000 08:04 1861222                    /usr/lib/python2.5/lib-dynload/_ctypes.so
7fa22ed48000-7fa22edae000 r-xp 00000000 08:04 1860433                    /usr/lib/python2.5/lib-dynload/unicodedata.so
7fa22edae000-7fa22efae000 ---p 00066000 08:04 1860433                    /usr/lib/python2.5/lib-dynload/unicodedata.so
7fa22efae000-7fa22efbd000 rw-p 00066000 08:04 1860433                    /usr/lib/python2.5/lib-dynload/unicodedata.so
7fa22efbd000-7fa22efc3000 r-xp 00000000 08:04 1861227                    /usr/lib/python2.5/lib-dynload/collections.so
7fa22efc3000-7fa22f1c2000 ---p 00006000 08:04 1861227                    /usr/lib/python2.5/lib-dynload/collections.so
7fa22f1c2000-7fa22f1c4000 rw-p 00005000 08:04 1861227                    /usr/lib/python2.5/lib-dynload/collections.so
7fa22f1c4000-7fa22f246000 rw-p 7fa22f1c4000 00:00 0 
7fa22f246000-7fa22f259000 r-xp 00000000 08:04 1861237                    /usr/lib/python2.5/lib-dynload/cPickle.so
7fa22f259000-7fa22f458000 ---p 00013000 08:04 1861237                    /usr/lib/python2.5/lib-dynload/cPickle.so
7fa22f458000-7fa22f45a000 rw-p 00012000 08:04 1861237                    /usr/lib/python2.5/lib-dynload/cPickle.so
7fa22f45a000-7fa22f45d000 r-xp 00000000 08:04 1860425                    /usr/lib/python2.5/lib-dynload/select.so
7fa22f45d000-7fa22f65d000 ---p 00003000 08:04 1860425                    /usr/lib/python2.5/lib-dynload/select.so
7fa22f65d000-7fa22f65e000 rw-p 00003000 08:04 1860425                    /usr/lib/python2.5/lib-dynload/select.so
7fa22f65e000-7fa22f69f000 rw-p 7fa22f65e000 00:00 0 
7fa22f69f000-7fa22f6b1000 r-xp 00000000 08:04 1861226                    /usr/lib/python2.5/lib-dynload/datetime.so
7fa22f6b1000-7fa22f8b0000 ---p 00012000 08:04 1861226                    /usr/lib/python2.5/lib-dynload/datetime.so
7fa22f8b0000-7fa22f8b4000 rw-p 00011000 08:04 1861226                    /usr/lib/python2.5/lib-dynload/datetime.so
7fa22f8b4000-7fa22f8f5000 rw-p 7fa22f8b4000 00:00 0 
7fa22f8f5000-7fa22f8f9000 r-xp 00000000 08:04 1860409                    /usr/lib/python2.5/lib-dynload/cStringIO.so
7fa22f8f9000-7fa22faf8000 ---p 00004000 08:04 1860409                    /usr/lib/python2.5/lib-dynload/cStringIO.so
7fa22faf8000-7fa22fafa000 rw-p 00003000 08:04 1860409                    /usr/lib/python2.5/lib-dynload/cStringIO.so
7fa22fafa000-7fa22fbbb000 rw-p 7fa22fafa000 00:00 0 
7fa22fbbb000-7fa22fbc9000 r-xp 00000000 08:04 1861236                    /usr/lib/python2.5/lib-dynload/parser.so
7fa22fbc9000-7fa22fdc9000 ---p 0000e000 08:04 1861236                    /usr/lib/python2.5/lib-dynload/parser.so
7fa22fdc9000-7fa22fdca000 rw-p 0000e000 08:04 1861236                    /usr/lib/python2.5/lib-dynload/parser.so
7fa22fdca000-7fa22fe4c000 rw-p 7fa22fdca000 00:00 0 
7fa22fe4c000-7fa22fe53000 r-xp 00000000 08:04 1860419                    /usr/lib/python2.5/lib-dynload/itertools.so
7fa22fe53000-7fa230053000 ---p 00007000 08:04 1860419                    /usr/lib/python2.5/lib-dynload/itertools.so
7fa230053000-7fa230056000 rw-p 00007000 08:04 1860419                    /usr/lib/python2.5/lib-dynload/itertools.so
7fa230056000-7fa230059000 r-xp 00000000 08:04 1860415                    /usr/lib/python2.5/lib-dynload/fcntl.so
7fa230059000-7fa230258000 ---p 00003000 08:04 1860415                    /usr/lib/python2.5/lib-dynload/fcntl.so
7fa230258000-7fa23025a000 rw-p 00002000 08:04 1860415                    /usr/lib/python2.5/lib-dynload/fcntl.so
7fa23025a000-7fa23025e000 r-xp 00000000 08:04 1861248                    /usr/lib/python2.5/lib-dynload/termios.so
7fa23025e000-7fa23045e000 ---p 00004000 08:04 1861248                    /usr/lib/python2.5/lib-dynload/termios.so
7fa23045e000-7fa230460000 rw-p 00004000 08:04 1861248                    /usr/lib/python2.5/lib-dynload/termios.so
7fa230460000-7fa230464000 r-xp 00000000 08:04 1860431                    /usr/lib/pyAborted




->> a interactive session


$ ./debug.py
Using cPO
Python-Levenshtein not found. Continuing with built-in (slower) fuzzy matching.

Welcome To Translate Toolkit Python Shell


>>> unit = storage.cpo.pounit("Test")
>>> unit.target = "Teste"
>>> unit
<translate.storage.cpo.pounit object at 0x7f482cb6c850>
>>> print unit
DEL CALLED
FREE CALLED
msgid "Test"
msgstr "Teste"


->> "prints" in __del__ and _free_iterator

>>> unit
<translate.storage.cpo.pounit object at 0x7f482cb6c850>
>>> unit.source
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/georgeyk/devel/translate/translate/storage/cpo.py", line 212, in getsource
    multi = multistring(singular, self._encoding)
  File "/home/georgeyk/devel/translate/translate/misc/multistring.py", line 35, in __new__
    newstring = autoencode.autoencode.__new__(newtype, string, encoding, errors)
  File "/home/georgeyk/devel/translate/translate/misc/autoencode.py", line 41, in __new__
    newstring = unicode.__new__(newtype, string, encoding)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xaf in position 2: unexpected code byte

->> This traceback makes sense, since the iterator has gone (in a sense that
->> it points to trash, but it is still in python heap)


>>> unit.target
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/georgeyk/devel/translate/translate/storage/cpo.py", line 251, in gettarget
    multi = multistring(gpo.po_message_msgstr(self._gpo_message) or u"", encoding=self._encoding)
  File "/home/georgeyk/devel/translate/translate/misc/multistring.py", line 35, in __new__
    newstring = autoencode.autoencode.__new__(newtype, string, encoding, errors)
  File "/home/georgeyk/devel/translate/translate/misc/autoencode.py", line 41, in __new__
    newstring = unicode.__new__(newtype, string, encoding)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 0: unexpected code byte
>>> print unit


->> this will call __del__ again

xerror_cb 0 <ctypes.LP_po_message object at 0xab0d98> None 0 0 1 The following msgctxt contains non-ASCII characters.
This will cause problems to translators who use a character encoding
different from yours. Consider using a pure ASCII msgctxt instead.
0.�

xerror_cb 0 <ctypes.LP_po_message object at 0xab0d98> None 0 0 1 The following msgid contains non-ASCII characters.
This will cause problems to translators who use a character encoding
different from yours. Consider using a pure ASCII msgid instead.
0.�

xerror_cb 1 <ctypes.LP_po_message object at 0xab0d98> None 0 0 0 invalid multibyte sequence
Traceback (most recent call last):
  File "/build/buildd/python2.5-2.5.2/Modules/_ctypes/callbacks.c", line 206, in 'calling callback function'
  File "/home/georgeyk/devel/translate/translate/storage/cpo.py", line 80, in xerror_cb
    raise ValueError(message_text)
ValueError: invalid multibyte sequence
xerror_cb 1 <ctypes.LP_po_message object at 0xcc05a0> None 0 0 0 invalid multibyte sequence
Traceback (most recent call last):
  File "/build/buildd/python2.5-2.5.2/Modules/_ctypes/callbacks.c", line 206, in 'calling callback function'
  File "/home/georgeyk/devel/translate/translate/storage/cpo.py", line 80, in xerror_cb
    raise ValueError(message_text)
ValueError: invalid multibyte sequence
Segmentation fault


->> comparing this with the the backtrace of the tests, it seems that we are
->> trying to release the memory twice



->> the first try:

Index: translate/storage/cpo.py
===================================================================
--- translate/storage/cpo.py	(revision 7967)
+++ translate/storage/cpo.py	(working copy)
@@ -669,16 +669,20 @@
     def __del__(self):
         # We currently disable this while we still get segmentation faults.
         # Note that this is definitely leaking memory because of this.
-        return
+        #return
+        print 'DEL CALLED'
         self._free_iterator()
         if self._gpo_memory_file is not None:
-            gpo.po_file_free(self._gpo_memory_file)
-            self._gpo_memory_file = None
+            del self._gpo_memory_file
+            #gpo.po_file_free(self._gpo_memory_file)
+            #self._gpo_memory_file = None

     def _free_iterator(self):
+        print 'FREE CALLED'
         # We currently disable this while we still get segmentation faults.
         # Note that this is definitely leaking memory because of this.
-        return
+        #return
         if self._gpo_message_iterator is not None:
-            gpo.po_message_iterator_free(self._gpo_message_iterator)
-            self._gpo_message_iterator = None
+            #gpo.po_message_iterator_free(self._gpo_message_iterator)
+            del self._gpo_message_iterator
+            #self._gpo_message_iterator = None



->> running tests



===================================================================== test process starts =====================================================================
executable:   /usr/bin/python  (2.5.2-final-0)
using py lib: /usr/lib/python2.5/site-packages/py <rev unknown>

test_cpo.py[58] ........F...........................................FF....

_______________________________________________________________________________________________________________________________________________________________
_______________________________________________________ entrypoint: TestCPOUnit().test_plural_reduction _______________________________________________________

    def test_plural_reduction(self):
        """checks that reducing the number of plurals supplied works"""
        unit = self.UnitClass("Tree")
        unit.msgid_plural = ["Trees"]
        assert isinstance(unit.source, multistring)
        assert unit.source.strings == ["Tree", "Trees"]
        unit.target = multistring(["Boom", "Bome", "Baie Bome"])
        assert isinstance(unit.source, multistring)
        assert unit.target.strings == ["Boom", "Bome", "Baie Bome"]
        unit.target = multistring(["Boom", "Bome"])
        assert unit.target.strings == ["Boom", "Bome"]
        unit.target = "Boom"
        # FIXME: currently assigning the target to the same as the first string won't change anything
        # we need to verify that this is the desired behaviour...
E       assert unit.target.strings == ["Boom", "Bome"]
>       assert [multistring([u'Boom'])] == ['Boom', 'Bome']
         +  where [multistring([u'Boom'])] = <translate.storage.cpo.pounit object at 0x7ff7a793abd0>.target.strings

[/home/georgeyk/devel/translate/translate/storage/test_cpo.py:51]
_______________________________________________________________________________________________________________________________________________________________
_________________________________________________ entrypoint: test_cpo.py TestCPOFile().test_merge_duplicates _________________________________________________

    def test_merge_duplicates(self):
        """checks that merging duplicates works"""
        posource = '#: source1\nmsgid "test me"\nmsgstr ""\n\n#: source2\nmsgid "test me"\nmsgstr ""\n'
        pofile = self.poparse(posource)
        #assert len(pofile.units) == 2
>       pofile.removeduplicates("merge")

[/home/georgeyk/devel/translate/translate/storage/test_po.py:446]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def removeduplicates(self, duplicatestyle="merge"):
        """make sure each msgid is unique ; merge comments etc from duplicates into original"""
        msgiddict = {}
        uniqueunits = []
        # we sometimes need to keep track of what has been marked
        # TODO: this is using a list as the pos aren't hashable, but this is slow...
        markedpos = []
        def addcomment(thepo):
            thepo.msgidcomment = " ".join(thepo.getlocations())
            markedpos.append(thepo)
        for thepo in self.units:
            if thepo.isheader():
                uniqueunits.append(thepo)
                continue
            if duplicatestyle.startswith("msgid_comment"):
                msgid = thepo._extract_msgidcomments() + thepo.source
            else:
                msgid = thepo.source
            if duplicatestyle == "msgid_comment_all":
                addcomment(thepo)
                uniqueunits.append(thepo)
            elif msgid in msgiddict:
                if duplicatestyle == "merge":
                    if msgid:
                        msgiddict[msgid].merge(thepo)
                    else:
                        addcomment(thepo)
                        uniqueunits.append(thepo)
                elif duplicatestyle == "keep":
                    uniqueunits.append(thepo)
                elif duplicatestyle == "msgid_comment":
                    origpo = msgiddict[msgid]
                    if origpo not in markedpos:
                        addcomment(origpo)
                    addcomment(thepo)
                    uniqueunits.append(thepo)
                elif duplicatestyle == "msgctxt":
                    origpo = msgiddict[msgid]
                    if origpo not in markedpos:
                        gpo.po_message_set_msgctxt(origpo._gpo_message, " ".join(origpo.getlocations()))
                        markedpos.append(thepo)
                    gpo.po_message_set_msgctxt(thepo._gpo_message, " ".join(thepo.getlocations()))
                    uniqueunits.append(thepo)
            else:
                if not msgid and duplicatestyle != "keep":
                    addcomment(thepo)
                msgiddict[msgid] = thepo
                uniqueunits.append(thepo)
        new_gpo_memory_file = gpo.po_file_create()
        new_gpo_message_iterator = gpo.po_message_iterator(new_gpo_memory_file, None)
        for unit in uniqueunits:
            gpo.po_message_insert(new_gpo_message_iterator, unit._gpo_message)
E       gpo.po_message_iterator_free(self._gpo_message_iterator)
>       AttributeError: 'pofile' object has no attribute '_gpo_message_iterator'

[/home/georgeyk/devel/translate/translate/storage/cpo.py:586]
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -  test_merge_duplicates: recorded stdout - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
DEL CALLED
FREE CALLED
FREE CALLED


->> probably the _free_iterator call at the end of the XXX method

- - - - - - - - - - - - - - - - - - - - - - - - - - - - -  test_merge_duplicates: recorded stderr - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
xerror2_cb 1 <ctypes.LP_po_message object at 0xbd20d8> /tmp/translateB79DqT.po 6 4294967295 0 duplicate message definition <ctypes.LP_po_message object at 0xbd2160> None 0 0 0 this is the location of the first definition
Exception exceptions.AttributeError: "'pofile' object has no attribute '_gpo_message_iterator'" in <bound method pofile.__del__ of <translate.storage.cpo.pofile object at 0xb79710>> ignored
Traceback (most recent call last):
  File "/build/buildd/python2.5-2.5.2/Modules/_ctypes/callbacks.c", line 206, in 'calling callback function'
  File "/home/georgeyk/devel/translate/translate/storage/cpo.py", line 85, in xerror2_cb
    raise ValueError(message_text1)
ValueError: duplicate message definition

_______________________________________________________________________________________________________________________________________________________________
_______________________________________________ entrypoint: test_cpo.py TestCPOFile().test_merge_mixed_sources ________________________________________________

    def test_merge_mixed_sources(self):
        """checks that merging works with different source location styles"""
        posource = '''
    #: source1
    #: source2
    msgid "test"
    msgstr ""
    
    #: source1 source2
    msgid "test"
    msgstr ""
    '''
        pofile = self.poparse(posource)
        print str(pofile)
>       pofile.removeduplicates("merge")

[/home/georgeyk/devel/translate/translate/storage/test_po.py:465]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def removeduplicates(self, duplicatestyle="merge"):
        """make sure each msgid is unique ; merge comments etc from duplicates into original"""
        msgiddict = {}
        uniqueunits = []
        # we sometimes need to keep track of what has been marked
        # TODO: this is using a list as the pos aren't hashable, but this is slow...
        markedpos = []
        def addcomment(thepo):
            thepo.msgidcomment = " ".join(thepo.getlocations())
            markedpos.append(thepo)
        for thepo in self.units:
            if thepo.isheader():
                uniqueunits.append(thepo)
                continue
            if duplicatestyle.startswith("msgid_comment"):
                msgid = thepo._extract_msgidcomments() + thepo.source
            else:
                msgid = thepo.source
            if duplicatestyle == "msgid_comment_all":
                addcomment(thepo)
                uniqueunits.append(thepo)
            elif msgid in msgiddict:
                if duplicatestyle == "merge":
                    if msgid:
                        msgiddict[msgid].merge(thepo)
                    else:
                        addcomment(thepo)
                        uniqueunits.append(thepo)
                elif duplicatestyle == "keep":
                    uniqueunits.append(thepo)
                elif duplicatestyle == "msgid_comment":
                    origpo = msgiddict[msgid]
                    if origpo not in markedpos:
                        addcomment(origpo)
                    addcomment(thepo)
                    uniqueunits.append(thepo)
                elif duplicatestyle == "msgctxt":
                    origpo = msgiddict[msgid]
                    if origpo not in markedpos:
                        gpo.po_message_set_msgctxt(origpo._gpo_message, " ".join(origpo.getlocations()))
                        markedpos.append(thepo)
                    gpo.po_message_set_msgctxt(thepo._gpo_message, " ".join(thepo.getlocations()))
                    uniqueunits.append(thepo)
            else:
                if not msgid and duplicatestyle != "keep":
                    addcomment(thepo)
                msgiddict[msgid] = thepo
                uniqueunits.append(thepo)
        new_gpo_memory_file = gpo.po_file_create()
        new_gpo_message_iterator = gpo.po_message_iterator(new_gpo_memory_file, None)
        for unit in uniqueunits:
            gpo.po_message_insert(new_gpo_message_iterator, unit._gpo_message)
E       gpo.po_message_iterator_free(self._gpo_message_iterator)
>       AttributeError: 'pofile' object has no attribute '_gpo_message_iterator'


[/home/georgeyk/devel/translate/translate/storage/cpo.py:586]
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -  test_merge_mixed_sources: recorded stdout - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
FREE CALLED
#: source1 source2
msgid "test"
msgstr ""


- - - - - - - - - - - - - - - - - - - - - - - - - - - - -  test_merge_mixed_sources: recorded stderr - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
xerror2_cb 1 <ctypes.LP_po_message object at 0xbd21e8> /tmp/translate8Zy0Mr.po 8 4294967295 0 duplicate message definition <ctypes.LP_po_message object at 0xbd2270> None 0 0 0 this is the location of the first definition
Traceback (most recent call last):
  File "/build/buildd/python2.5-2.5.2/Modules/_ctypes/callbacks.c", line 206, in 'calling callback function'
  File "/home/georgeyk/devel/translate/translate/storage/cpo.py", line 85, in xerror2_cb
    raise ValueError(message_text1)
ValueError: duplicate message definition

_______________________________________________________________________________________________________________________________________________________________
===================================================== tests finished: 55 passed, 3 failed in 0.19 seconds =====================================================




->> remove that call and run the tests again




===================================================================== test process starts =====================================================================
executable:   /usr/bin/python  (2.5.2-final-0)
using py lib: /usr/lib/python2.5/site-packages/py <rev unknown>

test_cpo.py[58] ........F.................................................

_______________________________________________________________________________________________________________________________________________________________
_______________________________________________________ entrypoint: TestCPOUnit().test_plural_reduction _______________________________________________________

    def test_plural_reduction(self):
        """checks that reducing the number of plurals supplied works"""
        unit = self.UnitClass("Tree")
        unit.msgid_plural = ["Trees"]
        assert isinstance(unit.source, multistring)
        assert unit.source.strings == ["Tree", "Trees"]
        unit.target = multistring(["Boom", "Bome", "Baie Bome"])
        assert isinstance(unit.source, multistring)
        assert unit.target.strings == ["Boom", "Bome", "Baie Bome"]
        unit.target = multistring(["Boom", "Bome"])
        assert unit.target.strings == ["Boom", "Bome"]
        unit.target = "Boom"
        # FIXME: currently assigning the target to the same as the first string won't change anything
        # we need to verify that this is the desired behaviour...
E       assert unit.target.strings == ["Boom", "Bome"]
>       assert [multistring([u'Boom'])] == ['Boom', 'Bome']
         +  where [multistring([u'Boom'])] = <translate.storage.cpo.pounit object at 0x7f7facb63a50>.target.strings

[/home/georgeyk/devel/translate/translate/storage/test_cpo.py:51]
_______________________________________________________________________________________________________________________________________________________________
===================================================== tests finished: 57 passed, 1 failed in 0.19 seconds =====================================================
DEL CALLED
FREE CALLED


->> better, but not enough. I need to measure the memory usage and the speed

Show details Hide details

Change log

r58 by georgeyk.dev on Aug 03, 2008 Diff

# Added mem_issue.txt: some thoughts about
the cpo memory leaks

Go to:

Older revisions

All revisions of this file

File info

Size: 27578 bytes, 507 lines

View raw file