Export to GitHub

sympy - issue #2587

Remove custom pickler from SymPy Live


Posted on Jul 22, 2011 by Happy Elephant

So, it turns out that this is a problem at live.sympy.org, not just locally on my machine:

>>> f = 1/(x**2*(x**2 + 1)) >>> f

1

2 / 1 \ x *|---- + 1| |/1 \ | ||--| | || 2| | \x / /

And similar for unicode. It prints right with LaTeX. And it works correctly in normal SymPy 0.7.0.

Comment #1

Posted on Jul 22, 2011 by Happy Elephant

Here's a more complete session:

f = 1/(x**2*(x**2 + 1)) f

1

2 / 1 \ x *|---- + 1| |/1 \ | ||--| | || 2| | \x / /

apart(f) 1 1 - ------ + -- 2 2 x + 1 x f 1
─────────── 2 ⎛ 2 ⎞ x ⋅⎝x + 1⎠ f

1

2 / 2 \ x *\x + 1/

Comment #2

Posted on Nov 26, 2011 by Happy Elephant

(No comment was entered for this change.)

Comment #3

Posted on Nov 27, 2011 by Massive Elephant

(No comment was entered for this change.)

Comment #4

Posted on Jan 6, 2012 by Happy Elephant

f = 1/(Catalan**2*(Catalan**2 + 1)) f

1

   2 /    1         \

Catalan *|---------- + 1| |/ 1 \ | ||--------| | || 2| | \Catalan / /

f == 1/(Catalan**2*(Catalan**2 + 1)) False

Note that I chose Catalan because it's a singleton. This is probably related to pickling (see issue 2204).

Comment #5

Posted on Jan 6, 2012 by Happy Elephant

This also shows that we need to use protocol 2 in the pickling. I tried this change, and it cause the above to create a traceback. I don't know if there were other problems, or if that was just this error manifesting itself in a different way.

Comment #6

Posted on Jan 6, 2012 by Happy Elephant

It is definitely pickling. In a regular IPython console with the git master sympy:

In [58]: f = 1/(x**2*(x**2 + 1))

In [63]: pickle.loads(pickle.dumps(f)) Out[63]: 1
───────────── 2 ⎛ 1 ⎞ x ⋅⎜──── + 1⎟ ⎜⎛1 ⎞ ⎟ ⎜⎜──⎟ ⎟ ⎜⎜ 2⎟ ⎟ ⎝⎝x ⎠ ⎠

In [64]: pickle.loads(pickle.dumps(f, -1)) Out[64]: 1
───────────── 2 ⎛ 1 ⎞ x ⋅⎜──── + 1⎟ ⎜⎛1 ⎞ ⎟ ⎜⎜──⎟ ⎟ ⎜⎜ 2⎟ ⎟ ⎝⎝x ⎠ ⎠

Comment #7

Posted on Jan 6, 2012 by Happy Elephant

Something global is being set. It may be related to the cache, but it must also be something else too. Here are some isympy sessions. I didn't change the numbering: these are the whole sessions. When the numbering restarts, it means it's a new session.

These are all with the cache off:

In [1]: import pickle

In [2]: f = 1/(x**2*(x**2 + 1))

In [3]: f Out[3]: 1
─────────── 2 ⎛ 2 ⎞ x ⋅⎝x + 1⎠

In [4]: g = pickle.loads(pickle.dumps(f))

In [5]: g Out[5]: 1
─────────── 2 ⎛ 2 ⎞ x ⋅⎝x + 1⎠

In [6]: f Out[6]: 1
─────────── 2 ⎛ 2 ⎞ x ⋅⎝x + 1⎠

In [1]: import pickle

In [2]: f = 1/(x**2*(x**2 + 1))

In [3]: g = pickle.loads(pickle.dumps(f))

In [4]: g Out[4]: 1
───────────── 2 ⎛ 1 ⎞ x ⋅⎜──── + 1⎟ ⎜⎛1 ⎞ ⎟ ⎜⎜──⎟ ⎟ ⎜⎜ 2⎟ ⎟ ⎝⎝x ⎠ ⎠

In [5]: f Out[5]: 1
─────────── 2 ⎛ 2 ⎞ x ⋅⎝x + 1⎠

In [6]: g Out[6]: 1
───────────── 2 ⎛ 1 ⎞ x ⋅⎜──── + 1⎟ ⎜⎛1 ⎞ ⎟ ⎜⎜──⎟ ⎟ ⎜⎜ 2⎟ ⎟ ⎝⎝x ⎠ ⎠

These are with the cache on:

In [1]: import pickle

In [2]: f = 1/(x**2*(x**2 + 1))

In [3]: f Out[3]: 1
─────────── 2 ⎛ 2 ⎞ x ⋅⎝x + 1⎠

In [4]: g = pickle.loads(pickle.dumps(f))

In [5]: g Out[5]: 1
─────────── 2 ⎛ 2 ⎞ x ⋅⎝x + 1⎠

In [6]: f Out[6]: 1
─────────── 2 ⎛ 2 ⎞ x ⋅⎝x + 1⎠

In [1]: import pickle

In [2]: f = 1/(x**2*(x**2 + 1))

In [3]: g = pickle.loads(pickle.dumps(f))

In [4]: g Out[4]: 1
───────────── 2 ⎛ 1 ⎞ x ⋅⎜──── + 1⎟ ⎜⎛1 ⎞ ⎟ ⎜⎜──⎟ ⎟ ⎜⎜ 2⎟ ⎟ ⎝⎝x ⎠ ⎠

In [5]: f Out[5]: 1
───────────── 2 ⎛ 1 ⎞ x ⋅⎜──── + 1⎟ ⎜⎛1 ⎞ ⎟ ⎜⎜──⎟ ⎟ ⎜⎜ 2⎟ ⎟ ⎝⎝x ⎠ ⎠

In [6]: g Out[6]: 1
───────────── 2 ⎛ 1 ⎞ x ⋅⎜──── + 1⎟ ⎜⎛1 ⎞ ⎟ ⎜⎜──⎟ ⎟ ⎜⎜ 2⎟ ⎟ ⎝⎝x ⎠ ⎠

Comment #8

Posted on Jan 6, 2012 by Happy Elephant

So I think the difference with the cache just means that it is using the cached printing of g for f. But the problem is something beyond this.

Comment #9

Posted on Jan 6, 2012 by Happy Elephant

This is the problem.

In [13]: g.args[1].args[0].args[1].args[1].is_negative Out[13]: True

This refers to the 2 exponent of the x**2 that is being printed wrong.

Comment #10

Posted on Jan 6, 2012 by Happy Elephant

This used to work correctly a long time ago. I bisected it to the commit aec5545e3b587, but I don't see what from that commit would cause it to do this.

Comment #11

Posted on Jan 6, 2012 by Happy Elephant

(No comment was entered for this change.)

Comment #12

Posted on Jan 6, 2012 by Happy Elephant

In [7]: a = g.args[1].args[0].args[1].args[1]

In [9]: b = f.args[1].args[0].args[1].args[1]

In [10]: b._assumptions Out[10]: {bounded: True, commutative: True, comparable: True, complex: True, finite: True, imaginary: False, infinitesimal: False, integer: True, irrational: False, negative: False, non integer: False, nonnegative: True, nonpositive: False, nonzero: True, positive: True, prime: None, rational: True, real: True, unbounded: False, zero: False}

In [8]: a._assumptions Out[8]: {bounded: True, commutative: True, comparable: True, complex: True, composite: False, finite: True, imaginary: False, infinitesimal: False, integer: True, irrational: False, ne gative: True, noninteger: False, nonnegative: False, nonpositive: True, nonzero: True, positive: False, prime: False, rational: True, real: True, unbounded: False, zero: False}

Assumptions are being computed incorrectly for pickled objects.

Comment #13

Posted on Jan 6, 2012 by Happy Elephant

This explains why the order matters. Assumptions are not computed until they are accessed. Remember that f is, internally, x**-2*(x**2 + 1). So it has a 2 and a -2. If you ask if the 2 is negative first, it gives the right answer, but comes out wrong for the 2:

n [1]: import pickle

In [2]: f = 1/(x**2*(x**2 + 1))

In [3]: g = pickle.loads(pickle.dumps(f))

In [4]: a = g.args[1].args[0].args[1].args[1]

In [5]: b = f.args[1].args[0].args[1].args[1]

In [6]: a.is_negative Out[6]: False

In [7]: b.is_negative Out[7]: False

In [8]: g Out[8]: -2
x
────── 2
x + 1

In [9]: g.args[0] Out[9]: -2 x

In [10]: g.args[0].args Out[10]: (x, -2)

In [11]: g.args[0].args[1] Out[11]: -2

In [12]: g.args[0].args[1].is_negative Out[12]: False

If you ask if the -2 is negative first, it comes out right, but then thinks the 2 is negative:

In [1]: import pickle

In [2]: f = 1/(x**2*(x**2 + 1))

In [3]: g = pickle.loads(pickle.dumps(f))

In [4]: g.args[0].args[1].is_negative Out[4]: True

In [5]: a = g.args[1].args[0].args[1].args[1]

In [6]: a.is_negative Out[6]: True

In [7]: g Out[7]: 1
───────────── 2 ⎛ 1 ⎞ x ⋅⎜──── + 1⎟ ⎜⎛1 ⎞ ⎟ ⎜⎜──⎟ ⎟ ⎜⎜ 2⎟ ⎟ ⎝⎝x ⎠ ⎠

In other words, it wants them both to be the same.

Comment #14

Posted on Jan 7, 2012 by Happy Giraffe

This is a simpler case that can reproduce the problem:

In [1]: from pickle import dumps,loads

In [2]: f=Add(Pow(x,2),Pow(x,-2)) # x**2+x**-2

In [3]: g=loads(dumps(f))

In [4]: g Out[4]: 2 -2 x + x

In [5]: f Out[5]: 2 1 x + -- 2 x

In [6]: f==g Out[6]: True

Comment #15

Posted on Jan 7, 2012 by Happy Giraffe

After further investigation it appears that the assumptions for the two exponents in x**2+x**-2 are pointing to the same object after pickling.

In [1]: from pickle import dumps, loads

In [2]: f = x**2+x**-2

In [3]: g = loads(dumps(f))

In [4]: f0 = f.args[0].args[1] # 2

In [5]: f1 = f.args[1].args[1] # -2

In [6]: g0 = g.args[0].args[1] # 2

In [7]: g1 = g.args[1].args[1] # -2

In [8]: f0.evalf() Out[8]: 2.00000000000000

In [9]: f1.evalf() Out[9]: -2.00000000000000

In [10]: g0.evalf() Out[10]: 2.00000000000000

In [11]: g1.evalf() Out[11]: -2.00000000000000

In [12]: f0.is_negative Out[12]: False

In [13]: f1.is_negative Out[13]: True

In [14]: g0.is_negative Out[14]: False

In [15]: g1.is_negative Out[15]: False

In [16]: f0._assumptions is f1._assumptions Out[16]: False

In [17]: g0._assumptions is g1._assumptions Out[17]: True

Comment #16

Posted on Jan 7, 2012 by Happy Giraffe

According to the pickle documentation [0], equivalent objects are combined and stored by reference instead of by value. This presents a problem in our case because the two exponents have the same assumptions when first created with x**2+x**-2. Pickle combines the two _assumptions objects into one, and it appears that it doesn't make copies when unpickling leading to the above problem. I'm investigating how to make copies when unpickling.

[0] - http://docs.python.org/library/pickle.html#pickle.Pickler

Comment #17

Posted on Jan 7, 2012 by Happy Giraffe

Looks like Pickler has a field "fast" for disabling the optimization. I wrote a fast_dumps function and it appears to work:

In [1]: from pickle import Pickler, loads

In [2]: from StringIO import StringIO

In [3]: def fast_dumps(obj, protocol=None): ...: file = StringIO() ...: p = Pickler(file, protocol) ...: p.fast = 1 ...: p.dump(obj) ...: return file.getvalue() ...:

In [4]: f = x**2 + x**-2

In [5]: g = loads(fast_dumps(f))

In [6]: f Out[6]: 2 1 x + -- 2 x

In [7]: g Out[7]: 2 1 x + -- 2 x

I'll go ahead and make a pull request with this fix in SymPy Live, though a note probably should be made somewhere in SymPy itself.

Comment #18

Posted on Jan 9, 2012 by Happy Elephant

Well, I looked into it, and not pickling _assumptions at all is not the answer. The problem is that, as far as I can tell, it's impossible to differentiate between assumptions that should be copied because they were set on an object (like Symbol('x', positive=True)), and those that can be recomputed from other information (like Integer(2).is_positive). Unless someone can come up with a clean solution on the SymPy side. Perhaps I'm missing something (e.g., is there a way to set inside SymPy to not optimize _assumptions in pickling?).

Since this would require a rewrite of the assumptions system, I think it's not worth it, as we plan on dumping the old assumptions system anyway. So for now, I think we should just go with your fix in SymPy Live. This needs to be tested more, but it's probably OK.

By the way, go ahead and use dumps(obj, -1) to use the highest protocol. This should be more efficient.

Other than that, we should add an XFAIL test for this in SymPy (sympy/utilities/tests/test_pickling.py), and perhaps document this somewhere.

Comment #19

Posted on Jan 14, 2012 by Happy Elephant

This was fixed at https://github.com/sympy/sympy-live/pull/46. See the discussion there. There are a lot of pickling problems with SymPy, and they all show up in SymPy Live.

Comment #20

Posted on Mar 6, 2012 by Happy Elephant

I think this issue should remain open so we can track the pickling problem.

Comment #21

Posted on Mar 20, 2012 by Happy Elephant

(No comment was entered for this change.)

Comment #22

Posted on May 7, 2012 by Happy Elephant

https://github.com/sympy/sympy/pull/1162 fixed this. I'm not sure what the implications are for SymPy Live.

Comment #23

Posted on May 7, 2012 by Happy Elephant

I guess it means we can remove the custom pickler created by the GCI student. And issue 2204 fixed means that we are no longer forced to use protocol 2, but we can use whatever protocol is the most efficient (this ought to be protocol 2, but tests by the GCI student seemed to reveal otherwise).

Comment #24

Posted on May 7, 2012 by Happy Elephant

(No comment was entered for this change.)

Comment #25

Posted on Oct 22, 2012 by Happy Elephant

(No comment was entered for this change.)

Comment #26

Posted on Oct 30, 2012 by Massive Dog

(No comment was entered for this change.)

Comment #27

Posted on Nov 8, 2012 by Happy Elephant

(No comment was entered for this change.)

Comment #28

Posted on Mar 5, 2014 by Happy Elephant

We have moved issues to GitHub https://github.com/sympy/sympy/issues.

Comment #29

Posted on Apr 6, 2014 by Happy Rabbit

Migrated to http://github.com/sympy/sympy/issues/5686

Status: Valid

Labels:
Type-Defect Priority-Medium Live Printing CodeInCategory-Code Assumptions CodeInImportedIntoGoogleDocs Restrict-AddIssueComment-Commit