| Issue 354: | python3 + rpy + pyodb + vertica 7 = Linux segfault | |
| 1 person starred this issue and may be notified of changes. | Back to list |
Hi Folks, I've discovered a python-vertica stack configuration that causes a segmentation fault on our Linux cluster. I'd really appreciate your help in deciding how to proceed with a solution. I've attached as much information as I can to help figure this out. The summary: When running in a virtualenv, if I import rpy2.objects then pyodb and then try to execute certain queries, I get the segfault. Interestingly, if I swap the import order, it seems to work OK (!) Testing on previous versions of pyodb have the segfault happening at connect() instead of execute(). Thanks very much, matt ** commands to install the stack via virtualenv /share/apps/python3/bin/python3.3 ~/bin/virtualenv.py --no-site-packages ~/virtualenvs/vertica-test source ~/virtualenvs/vertica-test/bin/activate pip install http://pyodbc.googlecode.com/files/pyodbc-3.0.7.zip -> Successfully installed pyodbc pip install rpy2 -> Successfully installed rpy2 ** commands in interpreter to reproduce the segfault # NB: bad username or password -> segfault at connect() # NB: if the import order is switched -> NO segfault [cornell@wright ~]$ source ~/virtualenvs/vertica-test/bin/activate (vertica-test)[cornell@wright ~]$ python3.3 Python 3.3.2 (default, May 16 2013, 11:31:48) [GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import rpy2.robjects >>> import pyodbc >>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UID=xx;PWD=xx') >>> curs = conn.cursor() >>> curs.execute('DROP TABLE IF EXISTS foo') Segmentation fault (vertica-test)[cornell@wright ~]$ python3.3 Python 3.3.2 (default, May 16 2013, 11:31:48) [GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import pyodbc >>> import rpy2.robjects >>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UID=xx;PWD=xx') >>> curs = conn.cursor() >>> curs.execute('DROP TABLE IF EXISTS foo') <pyodbc.Cursor object at 0x7f8508b5fe10> >>> ** versions Python 3.3.2 (default, May 16 2013, 11:31:48) [GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux >>> rpy2.__version__ '2.3.8' >>> pyodbc.version '3.0.7' >>> pyodbc.SQL_DBMS_VER 18 >>> pyodbc.SQL_DM_VER 171 >>> pyodbc.SQL_DRIVER_ODBC_VER 77 >>> pyodbc.SQL_DRIVER_VER 7 >>> pyodbc.SQL_ODBC_VER 10 $ isql --version unixODBC 2.2.14 /opt/vertica/lib64: lrwxrwxrwx 1 root root 23 Dec 18 09:43 libverticaodbc.so.7.0 -> libverticaodbc.so.7.0.0 [dbadmin@compute-0-0 ~]$ /opt/vertica/bin/adminTools Vertica Analytic Database 7.0.0-0 Administration Tools [cornell@wright ~]$ uname -a Linux wright.cs.umass.edu 2.6.32-279.14.1.el6.x86_64 #1 SMP Tue Nov 6 23:43:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux [cornell@wright ~]$ cat /proc/version Linux version 2.6.32-279.14.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Nov 6 23:43:09 UTC 2012 [cornell@wright ~]$ ** odbc configuration files *** /etc/odbcinst.ini [PostgreSQL] ... [MySQL] ... [vertica_driver] Description = Vertica 7 ODBC driver Driver = /opt/vertica/lib/libverticaodbc.so Driver64 = /opt/vertica/lib64/libverticaodbc.so FileUsage = 1 [ODBC] Threading = 1 Trace = 1 TraceFile = /tmp/odbctrace.log Debug = 1 DebugFile = /tmp/odbcdebug.log *** /etc/odbc.ini [ODBC Data Sources] vertica_kdl_dsn = kdl database on HP Vertica 7 [vertica_kdl_dsn] Driver = vertica_driver Description = ODBC Driver DSN for kdl master database Servername = compute-0-0 Port = 5433 Database = kdl Locale = en_US *** /etc/vertica.ini [Driver] DriverManagerEncoding = UTF-16 ODBCInstLib = /usr/lib64/libodbcinst.so ErrorMessagesPath = /opt/vertica/lib64 LogLevel = 4 LogPath = /tmp *** odbc trace files: see attached (wright-odbc-trace-2014-01-17) /tmp/odbctrace.log /tmp/vertica_odbc_conn_1.log /tmp/vertica_driver.log
Mar 9, 2014
Do you also experience this with rpy2-2.4.0-dev ?
Mar 31, 2014
Hi. Thanks for the question. I haven't tried it yet, but I'll try to get to it. Right now our workaround (swap the import order) is keeping us moving along. |
Here's some connection information, if that helps: <code> >>> import rpy2.robjects >>> import pyodbc >>> conn = pyodbc.connect('DSN=vertica_kdl_dsn;UserName=xx;Password=xx') >>> for attr in vars(pyodbc): ... try: ... value = conn.getinfo(getattr(pyodbc, attr)) ... print('{:<40s} | {}'.format(attr, value)) ... except: ... pass ... SQL_SQL_CONFORMANCE | 1 SQL_TXN_ISOLATION_OPTION | 10 SQL_PROCEDURE_TERM | function SQL_DROP_TRANSLATION | 0 SQL_CATALOG_USAGE | 0 SQL_SCHEMA_USAGE | 21 SQL_NULLABLE | 0 SQL_STATIC_CURSOR_ATTRIBUTES2 | 0 SQL_STATIC_CURSOR_ATTRIBUTES1 | 0 SQL_MAX_COLUMNS_IN_TABLE | 1600 SQL_GETDATA_EXTENSIONS | 15 SQL_INTERVAL_MONTH | 0 SQL_LIKE_ESCAPE_CLAUSE | True SQL_TIMEDATE_DIFF_INTERVALS | 511 SQL_ROW_UPDATES | False SQL_MAX_ROW_SIZE | 32000000 SQL_INTERVAL_YEAR | 1600 SQL_XOPEN_CLI_YEAR | 1995 SQL_INTERVAL_SECOND | 0 SQL_INTERVAL_MINUTE | 0 SQL_ACTIVE_ENVIRONMENTS | 0 SQL_MAX_INDEX_SIZE | 0 SQL_TIMEDATE_FUNCTIONS | 1998847 SQL_ALTER_TABLE | 62312 SQL_CREATE_TABLE | 1045 SQL_INTERVAL_DAY_TO_MINUTE | 511 SQL_SCOPE_SESSION | vertica_kdl_dsn SQL_SQL92_REVOKE | 13744 SQL_CREATE_TRANSLATION | 0 SQL_MAX_BINARY_LITERAL_LEN | 0 SQL_PARAM_INPUT | 0 SQL_PARAM_ARRAY_ROW_COUNTS | 1 SQL_COLUMN_ALIAS | True SQL_SQL92_ROW_VALUE_CONSTRUCTOR | 3 SQL_SCROLL_OPTIONS | 1 SQL_SCOPE_TRANSACTION | 0 lowercase | 0 SQL_BATCH_ROW_COUNT | 2 SQL_DYNAMIC_CURSOR_ATTRIBUTES2 | 0 SQL_DYNAMIC_CURSOR_ATTRIBUTES1 | 0 SQL_CATALOG_NAME | False SQL_SUBQUERIES | 31 SQL_SCHEMA_TERM | schema SQL_UNION | 3 SQL_NEED_LONG_DATA_LEN | False SQL_DATA_SOURCE_READ_ONLY | False SQL_DDL_INDEX | 0 SQL_NON_NULLABLE_COLUMNS | 1 SQL_MAX_IDENTIFIER_LEN | 128 SQL_QUOTED_IDENTIFIER_CASE | 4 SQL_SERVER_NAME | compute-0-0 SQL_UNKNOWN_TYPE | 0 SQL_NUMERIC | vertica_kdl_dsn SQLWCHAR_SIZE | vertica_kdl_dsn SQL_CORRELATION_NAME | 2 SQL_DESCRIBE_PARAMETER | False SQL_INTERVAL_HOUR_TO_MINUTE | False SQL_EXPRESSIONS_IN_ORDERBY | True SQL_OJ_CAPABILITIES | 127 SQL_TABLE_TERM | table SQL_SQL92_FOREIGN_KEY_DELETE_RULE | 0 SQL_CATALOG_LOCATION | 1 SQL_BOOKMARK_PERSISTENCE | 0 SQL_MAX_COLUMNS_IN_GROUP_BY | 0 SQL_CREATE_DOMAIN | 0 SQL_INTEGRITY | False SQL_DRIVER_NAME | verticaodbcw.so SQL_TYPE_TIME | 0 SQL_DEFAULT_TXN_ISOLATION | 2 SQL_INFO_SCHEMA_VIEWS | 0 SQL_TYPE_TIMESTAMP | 4 SQL_SCOPE_CURROW | 0 SQL_SQL92_NUMERIC_VALUE_FUNCTIONS | 46 SQL_ORDER_BY_COLUMNS_IN_SELECT | True SQL_TYPE_DATE | 21 SQL_CREATE_ASSERTION | 0 SQL_INTERVAL_MINUTE_TO_SECOND | True SQL_NULLABLE_UNKNOWN | vertica_kdl_dsn SQL_MULTIPLE_ACTIVE_TXN | True SQL_STRING_FUNCTIONS | 9732063 SQL_CONCAT_NULL_BEHAVIOR | 0 SQL_MAX_CATALOG_NAME_LEN | 0 SQL_MAX_SCHEMA_NAME_LEN | 128 SQL_MAX_CONCURRENT_ACTIVITIES | 0 SQL_MAX_CURSOR_NAME_LEN | 0 SQL_IDENTIFIER_CASE | 4 SQL_CURSOR_ROLLBACK_BEHAVIOR | 2 SQL_DBMS_VER | 07.00.0000 SQL_DROP_VIEW | 1 SQL_CHAR | 0 SQL_SEARCH_PATTERN_ESCAPE | \ SQL_REAL | 07.00.0000 SQL_GROUP_BY | 2 SQL_DATABASE_NAME | SQL_SQL92_GRANT | 3440 SQL_DROP_TABLE | 7 SQL_ACCESSIBLE_PROCEDURES | False SQL_BATCH_SUPPORT | 3 SQL_MAX_TABLES_IN_SELECT | 0 SQL_ODBC_VER | 03.52 SQL_COLLATION_SEQ | SQL_MAX_STATEMENT_LEN | 0 SQL_DRIVER_VER | 07.00.0000 SQL_DBMS_NAME | Vertica Database SQL_PARAM_INPUT_OUTPUT | vertica_kdl_dsn SQL_INTERVAL_HOUR_TO_SECOND | 0 SQL_PARAM_TYPE_UNKNOWN | 0 SQL_PC_UNKNOWN | 0 SQL_DM_VER | 03.52.0002.0002 SQL_CONVERT_VARCHAR | 12317183 SQL_DRIVER_ODBC_VER | 03.52 SQL_PC_NOT_PSEUDO | 0 SQL_INTERVAL_YEAR_TO_MONTH | 128 SQL_PC_PSEUDO | vertica_kdl_dsn SQL_INTERVAL_DAY | False SQL_FORWARD_ONLY_CURSOR_ATTRIBUTES2 | 4097 SQL_FORWARD_ONLY_CURSOR_ATTRIBUTES1 | 1 SQL_KEYSET_CURSOR_ATTRIBUTES2 | 4097 SQL_SQL92_STRING_FUNCTIONS | 255 SQL_CATALOG_TERM | SQL_INDEX_KEYWORDS | 0 SQL_CREATE_COLLATION | 0 SQL_AGGREGATE_FUNCTIONS | 127 SQL_FLOAT | verticaodbcw.so SQL_IDENTIFIER_QUOTE_CHAR | " SQL_MAX_TABLE_NAME_LEN | 128 SQL_MAX_USER_NAME_LEN | 128 SQL_SYSTEM_FUNCTIONS | 7 SQL_MAX_DRIVER_CONNECTIONS | 0 SQL_SQL92_PREDICATES | 16007 SQL_ODBC_INTERFACE_CONFORMANCE | 1 SQL_SQL92_FOREIGN_KEY_UPDATE_RULE | 0 SQL_MAX_ASYNC_CONCURRENT_STATEMENTS | 0 SQL_NUMERIC_FUNCTIONS | 16121855 SQL_INSERT_STATEMENT | 7 SQL_KEYSET_CURSOR_ATTRIBUTES1 | 4687 SQL_CREATE_SCHEMA | 3 SQL_SQL92_DATETIME_FUNCTIONS | 7 SQL_DROP_SCHEMA | 7 SQL_FILE_USAGE | 0 SQL_MAX_COLUMN_NAME_LEN | 128 SQL_KEYWORDS | SQL_PROCEDURES | False SQL_CREATE_CHARACTER_SET | 0 SQL_DROP_DOMAIN | 0 SQL_NULL_COLLATION | 1 SQL_STANDARD_CLI_CONFORMANCE | 2 SQL_TXN_CAPABLE | 3 SQL_SQL92_VALUE_EXPRESSIONS | 15 SQL_MAX_COLUMNS_IN_SELECT | 0 pooling | 0 SQL_MAX_CHAR_LITERAL_LEN | 0 SQL_ALTER_DOMAIN | 0 SQL_DROP_COLLATION | 0 SQL_PARAM_ARRAY_SELECTS | 3 SQL_CURSOR_COMMIT_BEHAVIOR | 2 SQL_SQL92_RELATIONAL_JOIN_OPERATORS | 474 SQL_MAX_ROW_SIZE_INCLUDES_LONG | False SQL_MAX_COLUMNS_IN_ORDER_BY | 0 SQL_INTERVAL_HOUR | 32000000 SQL_MAX_PROCEDURE_NAME_LEN | 128 SQL_MAX_COLUMNS_IN_INDEX | 0 SQL_CONVERT_FUNCTIONS | 3 SQL_CATALOG_NAME_SEPARATOR | . SQL_ACCESSIBLE_TABLES | False SQL_INTERVAL_DAY_TO_HOUR | 0 threadsafety | 0 SQL_SPECIAL_CHARACTERS | SQL_CREATE_VIEW | 1 SQL_INTERVAL_DAY_TO_SECOND | 511 SQL_DROP_CHARACTER_SET | 0 SQL_USER_NAME | test SQL_TIMEDATE_ADD_INTERVALS | 511 SQL_NO_NULLS | 0 SQL_DATETIME_LITERALS | 65535 SQL_DATA_SOURCE_NAME | vertica_kdl_dsn SQL_DROP_ASSERTION | 0 SQL_MULT_RESULT_SETS | True SQL_ASYNC_MODE | 0 >>> </code>