Issue 170: linux/freetds segfault with unicode strings
Status:  Complete
Owner: ----
Closed:  Oct 2011
Reported by rsyr...@gmail.com, Apr 7, 2011
Platform:
---------

Ubuntu 10.04 x64
Python 2.6.5
PyODBC 2.1.8 & also tried current git version
FreeTDS 0.82-1build6 (distro version) and also tried custom CVS build

Reproduce:
----------

Run the attached script on similar platform

What is the expected output?
----------------------------

'would commit, but segfault happens first'


What do you see instead?
------------------------

*** glibc detected *** /path/to/python/or/script: free(): invalid next size (fast): 0x0000000003e605b0 ***

or 

*** glibc detected *** /path/to/python/or/script: malloc(): memory corruption: 0x00000000026c26e0 ***

or rarely

python: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.


Other Info:
-----------

Note the comments in the attached file.  The other combinations of string length run fine.

Let me know if there is anything else I can do to help track this down.

test_pyodbc.py
773 bytes   View   Download
Apr 7, 2011
#1 rsyr...@gmail.com
TDS Dump when running the script is attached.
tdsdump.log
99 KB   View   Download
Apr 7, 2011
#2 rsyr...@gmail.com
ok, a little more info and a correction.  When running the test script against the distro's version of FreeTDS, I do not get a segfault.  Instead, I get the error:

pyodbc.Error: ('HY004', '[HY004] [FreeTDS][SQL Server]Invalid data type (0) (SQLBindParameter)


When I run the pyodbc tests (sqlservertests.py) against 0.82, then I get a lot of failures (output attached).  When I run them against my compiled version of freetds, they segfault.

test_errors.txt
22.1 KB   View   Download
Apr 7, 2011
#3 rsyr...@gmail.com
(SQLBindParameter) error was fixed by using a freshly compiled version of FreeTDS and setting ClientCharset=UTF-8 in the connection string.  Doing that makes the test script run without error.

However, pyodbc tests still segfault and I still get segfault problems when running unit tests on a web application I built using SQLAlchemy.
Apr 8, 2011
#4 rsyr...@gmail.com
Sorry for all the comments, I am continuing to try to figure this one out.  

I was able to prevent my application's unit tests from crashing by changing all my SQLAlchemy columns from String() to Unicode().  That lead me to be able to come up with a test script that reproduces the problem (attached) based on SQLAlchemy.  

In my attempt to translate that code to pure pyodbc (the test script previously attached), the error no longer shows up.  However, I'm pretty confident the bug is in pyodbc or freetds since pure python code shouldn't be able to crash python.

FreeTDS guys suggested I run valgrind.  So I did that:

valgrind --tool=memcheck -v --suppressions=valgrind-python.supp python test_freetds_unicode.py 2>valgrind-out.txt

based on the files here:

http://svn.python.org/projects/python/trunk/Misc/README.valgrind
http://svn.python.org/projects/python/trunk/Misc/valgrind-python.supp (I uncommented the parts they recommend uncommenting if you don't want to re-compile python)

However, when I run with valgrind, I don't get a segfault anymore!  Valgrind output is attached anyway, not sure how helpful it will be.

Let me know if there is anything else I can to help.


test_freetds_unicode.py
656 bytes   View   Download
valgrind-out.txt
133 KB   View   Download
May 20, 2011
Project Member #5 mkleehammer
This is worrisome: Use of uninitialised value of size 8

However, it seems to show up a lot.  Can you run valgrind on a simple Python script without pyodbc to see if it also generates these errors.  It might be an error in the Python runtime:

  class X(object): pass
  x = X()
  x.test = 1
  del x

Status: Investigating
Jun 6, 2011
#6 zzz...@gmail.com
Im getting this error when using Easysoft, not FreeTDS, drivers.   Im getting the ***glibc detected*** behavior on a fedora machine as well as an ubuntu machine - on a remote machine being tested by a colleague, he is getting the  "Invalid data type (0) (SQLBindParameter)" error.   I've never had this problem with FreeTDS.
Jun 7, 2011
#7 davidp.r...@gmail.com
I believe there is a bug in sqlwchar.cpp at line 154.  I think it should be len + 1 because sqlwchar_copy() includes the trailing null.

Also, the freeTDS 0.91 release seems to fix a lot of unicode related issues.

Oct 5, 2011
#8 asha...@chumpland.org
I am having this issue as well, and running original reporter's test_pyodbc.py script results in same errors. Error occurs on two different systems:

1. Gentoo x64, python 2.6.6, unixodbc 2.3.0, freetds 0.83dev (from 2011-01-25), pyodbc 2.1.8
2. Gentoo x64, python 2.7.1, unixodbc 2.3.0, freetds 0.91, pyodbc 2.1.11

Oct 26, 2011
#9 alex.s.y...@gmail.com
Thank you, davidp.r...@gmail.com (comment #7)!

Your suggestion with len in sqlwchar.cpp at line 154 helped me (see attached patch).

OpenSUSE 11.1, Python 2.6.2, pyodbc 2.1.11, freetds 0.91.
pyodbc_sqlwchar.patch
401 bytes   View   Download
Oct 27, 2011
Project Member #10 mkleehammer
I am very grateful for you guys finding this.  Fixed in 2.1.13-beta01 and py3-3.0.1-beta06.

Status: Complete
Nov 5, 2011
#11 mike.dob...@gmail.com
Thank you for that patch. I hope to see it soon in stable version.