My favorites | Sign in
Project Home Downloads Wiki
READ-ONLY: This project has been archived. For more information see this post.
Search
for
  Advanced search   Search tips   Subscriptions
Issue 380: appcrash when dealing with large number of long varchar fields
1 person starred this issue and may be notified of changes. Back to list
Status:  New
Owner:  ----


 
Reported by gall.a...@gmail.com, Aug 21, 2014
What steps will reproduce the problem?
1. Connect to Filemaker Pro database (I am using Filemaker pro 13, driver version 13.02.14 but have experienced this issue across several different versions of filemaker pro and filemaker pro drivers)

cnxn = pyodbc.connect(Driver='FileMaker ODBC',Server='localhost',DSN='DSN',database='DBASE',UID='UID',PWD='PWD')

2. iterate over a large number of long varchar records. I have tried simplifying down to the following where:

len(articles.article) < 40,000 but typically greater than 2k 
and 
the article table has appx 10k records:

cursor.execute("SELECT article FROM articles")
for row in cursor:
    temp = articles.article

I have also tried iterating over the data set in chunks of various sizes. ie. SELECT first 20 articles... next 20... etc. iterating over a small SELECT usually works better but still crashes. Also using an unordered SELECT usually performs better than an ordered select.


What is the expected output? What do you see instead?

I expect to be able to fetch the records without issue or at least throw an error but instead always get appcrash. Also app crash does not happen at a specific record but randomly. ie. sometimes will crash on the 500th record, sometimes on the two-thousandth. Sometimes on an extra long record, sometimes on a short record.


What version of the product are you using? On what operating system?
anaconda python 2.7 32bit (have also tried on 64bit and vanilla python)
pyodbc - various versions up to the latest commit as of aug 21, 2014.
various versions of the filemaker odbc driver. currently 13.02.14
Filemaker Pro 13 though have also tried on 11, 12 and FMP Server.


Please provide any additional information below.
I am able to successfully execute this query in Microsoft excel so it seems that the error is being caused by pyodbc and not necessarily the filemaker pro driver.

Here is a sample ODBC trace where the app crashes. In this instance it crashed when processing a short record.

python          570-644	ENTER SQLFetch 
		HSTMT               0x0306E6E0

python          570-644	EXIT  SQLFetch  with return code 0 (SQL_SUCCESS)
		HSTMT               0x0306E6E0

python          570-644	ENTER SQLGetData 
		HSTMT               0x0306E6E0
		UWORD                        1 
		SWORD                        1 <SQL_C_CHAR>
		PTR                 0x0027F7B6 
		SQLLEN                  1024
		SQLLEN *            0x0027F790

python          570-644	EXIT  SQLGetData  with return code 0 (SQL_SUCCESS)
		HSTMT               0x0306E6E0
		UWORD                        1 
		SWORD                        1 <SQL_C_CHAR>
		PTR                 0x0027F7B6 [     559] "...some text..."
		SQLLEN                  1024
		SQLLEN *            0x0027F790 (559)

python          570-644	ENTER SQLFetch 
		HSTMT               0x0306E6E0


Aug 21, 2014
#1 gall.a...@gmail.com
error in code above... should read: 

for row in cursor:
    temp = row.article
Aug 21, 2014
#2 gall.a...@gmail.com
update: if I increase the buffer size by changing line 330 of getdata.cpp from 

char tempBuffer[1026]; // Pad with 2 bytes for driver bugs

to

char tempBuffer[96002]; // Pad with 2 bytes for driver bugs

The issue seems to be resolved. Though I cant imagine that setting the buffer so large is the desired behavior.

Powered by Google Project Hosting