QSqlQuery hangs/crashes with QODBC driver - ODBC driver verified working with Python

jars121

Hi all,

I'm trying to connect to a SQLBase database using its ODBC driver. I'm on Windows 10 using Qt 6.6.0 with MinGW and qmake. I have the following Python script which works perfectly:

import pyodbc
import pandas as pd 
import os.path

def sql_connection(): 
    return pyodbc.connect('Driver=Gupta SQLBase 12.3 (64-bit);' + 'Server=127.0.0.1' + ';ServerName=SERVER1' + ';Port=2155' + ';Database=DB1' + ';UID=USER1' + ';PWD=TEST1;')

if __name__ == '__main__':

        sql_text = "SELECT * FROM TABLE1"
        
        df = pd.read_sql(sql_text,sql_connection())
        
        with pd.option_context('display.max_rows', None, 'display.max_columns', None):
            print(df)

This simple Python script uses the SQLBase ODBC driver to return all the data from TABLE1. This works as expected, returning 118 results. The results are correct and have been validated by using the SQLTalk Plus client using the native SQLBase driver.

I'm now trying to replicate this basic functionality in my Qt application, but am finding some strange results. Take the following basic example:

QSqlDatabase testDatabase = QSqlDatabase::addDatabase("QODBC");

testDatabase.setDatabaseName("Driver=Gupta SQLBase 12.3 (64-bit);Server=127.0.0.1;ServerName=SERVER1;Database=DB1;Port=2155;UID=USER1;PWD=TEST1");

if (!testDatabase.open()) {
    qDebug() << "Error opening database.";
}

QSqlQuery testQuery = QSqlQuery("SELECT * FROM TABLE1", testDatabase);

while (testQuery.next()) {

    qDebug() << testQuery.record().value(0) << testQuery.record().value(1);
}

The connection to the database and opening of the database works correctly. However, rather than printing results, the application hangs for quite some time (several minutes), during which time I can see the memory used by the application steadily increasing. The application eventually crashes with the following:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I will note that if I change the select statement to reduce the number of returned results (for example 20 instead of the full return of 118), the results and printed instantly and the application runs as expected. For example, this does work:

QSqlDatabase testDatabase = QSqlDatabase::addDatabase("QODBC");

testDatabase.setDatabaseName("Driver=Gupta SQLBase 12.3 (64-bit);Server=127.0.0.1;ServerName=SERVER1;Database=DB1;Port=2155;UID=USER1;PWD=TEST1");

if (!testDatabase.open()) {
    qDebug() << "Error opening database.";
}

QSqlQuery testQuery = QSqlQuery("SELECT * FROM TABLE1 WHERE TEST_DATE > &DATEVALUE('2023-01-01')", testDatabase);

while (testQuery.next()) {

    qDebug() << testQuery.record().value(0) << testQuery.record().value(1);
}

What might be causing this? It looks like the SQLBase ODBC driver works correctly, as I'm able to use it with pyodbc in Python. The fact that the process works for a limited returned data set suggests that the QODBC interface is partially working, but it's running into a memory leak/loop when a certain threshold is breached.

jars121

Quick update to the above. I've done some further debugging and have found that the QSqlQuery itself is working as expected, it's the QSqlRecord which is causing the observed hang/memory leak.

If I change the while() loop to the below, the application executes correctly:

int i = 0;
while (testQuery.next()) {
    qDebug() << ++i;
}

The numbers 1 through 118 are printed as expected and the application works correctly. However, if I try and access the QSqlRecord data either by directly instantiating a QSqlRecord object or via the QSqlQuery object itself, the hang/memory leak and crash is observed:

while (testQuery.next()) {
    QSqlRecord rec = testQuery.record();
    qDebug() << rec.value(0);
    //qDebug() << testQuery.record().value(0);
}

jars121

Another quick update.

I've just installed SAP's odbc-cpp-wrapper and linked it into my application.

I've replicated the above simple example as follows:

#include <iostream>
#include <odbc/Connection.h>
#include <odbc/Environment.h>
#include <odbc/Exception.h>
#include <odbc/PreparedStatement.h>
#include <odbc/ResultSet.h>

void testFunction() {
    try {

        odbc::EnvironmentRef env = odbc::Environment::create();
        odbc::ConnectionRef conn = env->createConnection();
        conn->connect("Driver=Gupta SQLBase 12.31 (64-bit);Server=127.0.0.1;ServerName=SERVER1;Database=DB1;Port=2155;UID=USER1;PWD=TEST1");
        conn->setAutoCommit(false);

        odbc::PreparedStatementRef psSelect = conn->prepareStatement("SELECT * FROM TABLE1");
        odbc::ResultSetRef rs = psSelect->executeQuery();
        while (rs->next())
        {
            qDebug() << rs->getNString(1)->c_str();
        }
    }
    catch (const odbc::Exception &e) {
        qDebug() << e.what();
    }
}

The above correctly prints the 118 ID strings from the first column of TABLE1 as expected. As such, it looks like there is a bug in QSqlRecord.

Christian Ehrlicher

Please provide a minimal, compilable example to reproduce the issue including the creation of the data in the code.
What database do you use?
Also you should try with QSqlQuery::setForwardOnly() and don't go through record() and print the single values with qDebug() to see where exactly it is hanging. A backtrace would also be nice.

jars121

@Christian-Ehrlicher Thank you for your prompt response Christian. I've just verified the issue on a brand new project; here is the .pro file:

QT = core sql

CONFIG += c++17 cmdline

# You can make your code fail to compile if it uses deprecated APIs.
# In order to do so, uncomment the following line.
#DEFINES += QT_DISABLE_DEPRECATED_BEFORE=0x060000    # disables all the APIs deprecated before Qt 6.0.0

SOURCES += \
        main.cpp

# Default rules for deployment.
qnx: target.path = /tmp/$${TARGET}/bin
else: unix:!android: target.path = /opt/$${TARGET}/bin
!isEmpty(target.path): INSTALLS += target

And the main.cpp file:

#include <QCoreApplication>

#include <QSqlDatabase>
#include <QSqlQuery>
#include <QSqlRecord>
#include <QSqlError>
#include <QDebug>

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    QSqlDatabase testDatabase = QSqlDatabase::addDatabase("QODBC");
    testDatabase.setDatabaseName("Driver=Gupta SQLBase 12.3 (64-bit);Server=127.0.0.1;ServerName=SERVER1;Database=DB1;Port=2155;UID=USER1;PWD=TEST1");

    if (!testDatabase.open()) {

        qDebug() << testDatabase.lastError();
    }

    QSqlQuery testQuery(testDatabase);
    testQuery.setForwardOnly(true);

    testQuery.prepare("SELECT * FROM TABLE1");
    testQuery.exec();

    while (testQuery.next()) {

        //Both of the below qDebug() statements cause the issue
        qDebug() << testQuery.value(0).toString();
        //qDebug() << testQuery.record().value(0);
    }

    return a.exec();
}

The database is SQLBase version 12.3. As shown above, I've tried using setForwardOnly() (I tried that earlier but didn't include it in my earlier posts) but the issue remains.

I put a break point on the qDebug() statement in the while() loop and created the following backtrace:

Thread 1 (Thread 32952.0x134c):
#0  0x00007ffc01911a10 in ntdll!RtlCompareMemoryUlong () from C:\WINDOWS\SYSTEM32\ntdll.dll
No symbol table info available.
#1  0x00007ffc0189dd00 in ntdll!RtlAllocateHeap () from C:\WINDOWS\SYSTEM32\ntdll.dll
No symbol table info available.
#2  0x00007ffc0189b44d in ntdll!RtlAllocateHeap () from C:\WINDOWS\SYSTEM32\ntdll.dll
No symbol table info available.
#3  0x00007ffc019688d8 in ntdll!RtlRegisterSecureMemoryCacheCallback () from C:\WINDOWS\SYSTEM32\ntdll.dll
No symbol table info available.
#4  0x00007ffc0189d255 in ntdll!RtlAllocateHeap () from C:\WINDOWS\SYSTEM32\ntdll.dll
No symbol table info available.
#5  0x00007ffc0189b44d in ntdll!RtlAllocateHeap () from C:\WINDOWS\SYSTEM32\ntdll.dll
No symbol table info available.
#6  0x00007ffc011a9d40 in msvcrt!malloc () from C:\WINDOWS\System32\msvcrt.dll
No symbol table info available.
#7  0x00007ffb302818c0 in QArrayData::allocate(QArrayData**, long long, long long, long long, QArrayData::AllocationOption) () from C:\Qt\6.6.0\6.6.0\mingw_64\bin\Qt6Core.dll
No symbol table info available.
#8  0x00007ffb30249303 in QString::QString(long long, Qt::Initialization) () from C:\Qt\6.6.0\6.6.0\mingw_64\bin\Qt6Core.dll
No symbol table info available.
#9  0x00007ffb3025e110 in QStringDecoder::decoderForHtml(QByteArrayView) () from C:\Qt\6.6.0\6.6.0\mingw_64\bin\Qt6Core.dll
No symbol table info available.
#10 0x00007ffb302447e1 in QString::fromUtf8(QByteArrayView) () from C:\Qt\6.6.0\6.6.0\mingw_64\bin\Qt6Core.dll
No symbol table info available.
#11 0x00007ffbdb51af49 in qt_plugin_instance () from C:\Qt\6.6.0\6.6.0\mingw_64\plugins\sqldrivers\qsqlodbc.dll
No symbol table info available.
#12 0x00007ffbdb51b285 in qt_plugin_instance () from C:\Qt\6.6.0\6.6.0\mingw_64\plugins\sqldrivers\qsqlodbc.dll
No symbol table info available.
#13 0x00007ffbdb51d8f2 in qt_plugin_instance () from C:\Qt\6.6.0\6.6.0\mingw_64\plugins\sqldrivers\qsqlodbc.dll
No symbol table info available.
#14 0x00007ffbdf26c0e8 in QSqlQuery::value(int) const () from C:\Qt\6.6.0\6.6.0\mingw_64\bin\Qt6Sql.dll
No symbol table info available.
#15 0x00007ff7bce917a8 in main (argc=1, argv=0x1f74d113c10) at ..\SQLBaseTest\main.cpp:30
        a = <incomplete type>
        testDatabase = {static defaultConnection = 0x7ffbdf2992ca "qt_sql_default_connection", d = 0x1f74d116100}
        __PRETTY_FUNCTION__ = "int main(int, char**)"
        testQuery = {d = 0x1f74d11c140}

Christian Ehrlicher

As written in the bug report please also create the table and data in the code. I would guess there is garbage in the database.

jars121

The database is sourced from a third party; it's not created/controlled by myself or my code base.

I have verified the data in the database with both Python and the odbc-cpp-wrapper library without issue. I've also viewed the data using the SQLTalk Plus client software provided by SQLBase.

This particular database is used extensively elsewhere in our environment, this is just the first time interfacing with it using Qt.

If it will help with diagnosis I can create a new database and dummy table and populate it with dummy data for testing. Unfortunately access to the SQLBase driver requires registration (as far as I can recall) and installation and setup of the server isn't particularly intuitive.

Christian Ehrlicher

You should extend your testcase to create and populate your table.

jars121

I'll endeavour to put together CREATE and INSERT code as well.

In the meantime I've just tested with Qt 5.15.10 (also MinGW with qmake on Windows 10) and it works perfectly. I haven't checked the release notes but it looks like perhaps something in the Qt5 -> Qt6 transition may be to blame.

ChrisW67

@jars121 Change "SELECT * ..." to SELECT column1, column2 ...", selecting only those you need. Querying by name is good practice for a number of reasons. Then, things I would try:

Rebuild any index on the table to eliminate corrupted indexes.
Try the query with a WHERE condition that bisects the data (using a key in some way) to isolate a trigger row if there is one. It looks like any row where TEST_DATE is in this year is suspect. That might be one row that you can already pinpoint, but it may also be a red herring.
Try the query one column at a time to isolate a trigger column if there is one. IMHO Columns most likely to break things would be dates/times or blobs.

jars121

@ChrisW67 I'll give this a go shortly, thanks for your input.

I do find it strange that the 'worst case' example code above (i.e. SELECT *) works perfectly fine on 5.15.10 though (as well as with Python and the odbc-cpp-wrapper library); if there is a trigger causing issues it seems to only trip up 6.6.0.

ChrisW67

@jars121 said in QSqlQuery hangs/crashes with QODBC driver - ODBC driver verified working with Python:

if there is a trigger causing issues it seems to only trip up 6.6.0.

You only know it trips up the Qt 6 version you have tested. Others may or may not work. Until the specific data that trips it up is isolated or the actual data is eliminated as the cause, little progress will happen on your QTBUG-119753. You have been asked for more information here and in the ticket for that reason.

Christian Ehrlicher

@jars121 said in QSqlQuery hangs/crashes with QODBC driver - ODBC driver verified working with Python:

if there is a trigger causing issues it seems to only trip up 6.6.0.

When I'm correct it will also not work with the latest Qt5.15 (lts) and Qt6.5/6.2. If so then you have to blame the odbc driver since it does not return the correct string lengths. There was a fix for it to follow the odbc standard (and avoid wrong usage) but some ancient odbc drivers are not following the standard. So - please provide a fully working example.