Export to GitHub

django-pyodbc - issue #91

Problems with BigInts.


Posted on Oct 18, 2010 by Happy Lion

What steps will reproduce the problem? 1. Fire up django shell 2. Import models 3. Filter results for a known existing bigint ID and it fails to return the row or test equality on the bigint field produces an error and renders the cursor useless.

What is the expected output? What do you see instead?

A single filtered row when either filtering or when comparing equality.

Here's what I see instead when filtering:

>>> from trades.models import BrokerQuoteTrades >>> BrokerQuoteTrades.objects.all() [<BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, '...(remaining elements truncated)...'] >>> BrokerQuoteTrades.objects.filter(id=634226759876817143) [] >>> trade = BrokerQuoteTrades.objects.filter(quoteid=320179)[0] >>> trade.id 634226759876817143L >>> BrokerQuoteTrades.objects.filter(id=long(634226759876817143)) []

And here's what I get when comparing the IDs:

>>> from trades.models import BrokerQuoteTrades >>> BrokerQuoteTrades.objects.all() [<BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, <BrokerQuoteTrades: BrokerQuoteTrades object>, '...(remaining elements truncated)...'] >>> trade = [trade for trade in BrokerQuoteTrades.objects.all() if trade.id == 634226759876817143] Traceback (most recent call last): File "<console>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 106, in _result_iter self._fill_cache() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 760, in _fill_cache self._result_cache.append(self._iter.next()) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 269, in iterator for row in compiler.results_iter(): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 672, in results_iter for rows in self.execute_sql(MULTI): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 747, in execute_sql return list(result) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 741, in <lambda> result = iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)), File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/django_pyodbc/sql_server/pyodbc/base.py", line 350, in fetchmany return [self.format_results(row) for row in self.cursor.fetchmany(chunk)] File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/django_pyodbc/sql_server/pyodbc/base.py", line 338, in format_results fr.append(row.decode('utf-8')) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError: 'utf8' codec can't decode byte 0xa3 in position 137: invalid start byte >>> BrokerQuoteTrades.objects.all() Traceback (most recent call last): File "<console>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 68, in repr data = list(self[:REPR_OUTPUT_SIZE + 1]) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 83, in len self._result_cache.extend(list(self._iter)) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/query.py", line 269, in iterator for row in compiler.results_iter(): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 672, in results_iter for rows in self.execute_sql(MULTI): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 727, in execute_sql cursor.execute(sql, params) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/util.py", line 15, in execute return self.cursor.execute(sql, params) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/db/backends/django_pyodbc/sql_server/pyodbc/base.py", line 313, in execute return self.cursor.execute(sql, params) ProgrammingError: ('24000', '[24000] [FreeTDS][SQL Server]Invalid cursor state (0) (SQLExecDirectW)')

What version of the product are you using? On what operating system?

python-2.7 freetds-0.82.1.dev.20100810 django-1.2.3 django-pyodbc-r183 Mac OS 10.6.4

Please provide any additional information below.

This works when running the SQL generated from the debug info directly from the django shell or when using pyodbc...

Comment #1

Posted on Oct 18, 2010 by Happy Lion

After going through the other issues on this list, a workaround was published: (issue 64)

http://code.google.com/p/django-mssql/source/browse/trunk/source/sqlserver_ado/fields.py

Any idea when a real fix will come?

Comment #2

Posted on Mar 24, 2011 by Swift Panda

(No comment was entered for this change.)

Comment #3

Posted on Jun 5, 2012 by Happy Wombat

I have the same problem. I've racked my brains for a several days investigating the problem deeper and deeper. The problem in preparing the where clause at django/db/models/sql/where.py:328 in make_atom() function when in casts value from child object.

And now I got simple solution - to use TextField in your model, while still bigint in DB. It works perfectly for me.

Status: Thinking

Labels:
Type-Defect Priority-Medium