QThread with SQLite blocks GUI

[Niagarer]

Hi,
in my program, a second thread, meant to extract data from one database and store it in another, gets created.

from mainwindow.cpp

void MainWindow::processFile(){
    if(unfinishedFilenames->length() > 0){
        QString path = unfinishedFilenames->at(0);
        QString filename = path.mid(path.lastIndexOf("/")+1);
        qDebug() << "processing file: " << filename;

        CurrentWeatherWorker *cww = new CurrentWeatherWorker(path, filename, targetDir);
        connect(cww, SIGNAL(fileProcessingFinished(QString)),
                this, SLOT(fileProcessingFinished(QString)));
        connect(cww, SIGNAL(updateProgress(int)),
                this, SLOT(updateProgressBar(int)));
        connect(cww, SIGNAL(log(QString)),
                this, SLOT(log(QString)));
        cww->start();
    }

}

currentweatherworker.cpp

#include "CurrentWeatherWorker.h"

#include "dataobjects.h"

#include <QJsonDocument>
#include <QJsonObject>
#include <QJsonArray>
#include <QtSql/QSqlDatabase>
#include <QSqlQuery>
#include <QSqlError>
#include <QVariant>


CurrentWeatherWorker::CurrentWeatherWorker(QString _filepath, QString _filename, QString _targetDir)
{
    filepath = _filepath;
    filename = _filename;
    targetDir = _targetDir;
}

void CurrentWeatherWorker::run(){
    log("opening database...");

    QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE");
    db.setDatabaseName(filepath);
    if(!db.open()){
        return;
    }

    QSqlQuery query;

    QList<CurrentData_meta> metaList;

    int numRecords = -1;
    query.exec("SELECT ID FROM Data");
    query.last();
    numRecords = query.value("ID").toInt();

    query.exec("SELECT * FROM Data");
    int i=0;
    while(query.next()){
        CurrentData_meta r;
        r.key = query.value("Key").toString();
        // ...

        metaList.append(r);

        log("data object read");

        emit updateProgress((100*i++)/(100*numRecords));
    }

    QString day = "";
    for(int i=9; i<filename.length(); i++){
        day+=filename.at(i);
    }
    QString newFilename = targetDir+"meta_data_current_"+day;

    saveMetaData_currentWeather(metaList, newFilename);

    emit updateProgress(100);

    emit fileProcessingFinished(filepath);
}


void CurrentWeatherWorker::saveMetaData_currentWeather(QList<CurrentData_meta> metaList, QString filename){

    log("creating new database...");

    QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE");

    db.setDatabaseName(filename);
    if(!db.open()){
        return;
    }

    QSqlQuery query;

    // creating table

    // preparing query
    for(int i=0; i<metaList.length(); i++){
        CurrentData_meta r = metaList[i];

        query.bindValue(":Key", r.key);
        // ...
        query.exec();

        QString data = "";
        QMap<QString, QVariant> map = query.boundValues();
        for(auto e : map.keys()){
            data += e + ": " + map.value(e).toString() + "; ";
        }

        log("data inserted: "+data);

        emit updateProgress((100*i)/(100*metaList.length()));
    }
    db.close();
    log("successfully finished saving data");
}

The worker works, but blocks the mainwindow (program not responding is shown by Windows) while it is running. Normalls this is due to a dependency between the two classes, which is why I use Qt signals an slots for communication, nothing else and I can't find another. The dataobjects.h file does not include anything. Sometimes, the mainwindow responds again but just for a couple of seconds (which is strange), then it's gone again.
Do you see something causing a connection between MainWindow and CurrentWeatherWorker responsible for the block? Does SQLite have something to do with it?
Thanks for answers!

[CKurdu]

@Niagarer said in QThread with SQLite blocks GUI:

log("opening database...");

Because I don't see where this log implemented, I am guessing this system-wide log is responsible for that. You can try to replace log calls like below.

emit  log("data object read parsed successfully");

[Niagarer]

@CKurdu
Oh, thank you, didn't see that!
Unfortunately, it still blocks the GUI.

[CKurdu]

@Niagarer said in QThread with SQLite blocks GUI:

emit updateProgress((100i++)/(100numRecords));

Maybe this line the reason. Because you are sending many update progress messages to the main thread. Maybe you can implement like below code in two places ( run and saveMetaData_currentWeather methods)

int progress = 0;
const int range = 100;
const int step = int(numRecords/range);
while(query.next())
{
//... some codes
if( (i%step )== 0 ) 
{
     progress++;
     emit updateProgress(progress);
}
}

[Christian Ehrlicher]

QSqlQuery query;

This is wrong when you're more than one db connection - you want 'QSqlQuery query(db)'

...saveMetaData_currentWeather()
...
QSqlDatabase db = QSqlDatabase::addDatabase("QSQLITE");

Here you're opening the database again and kill the old connection.

CurrentWeatherWorker *cww;

cww is leaking...

Apart from this I don't see what should block the ui thread apart from the stuff @CKurdu already mentioned.

[Niagarer]

@CKurdu said in QThread with SQLite blocks GUI:

Because you are sending many update progress messages to the main thread. Maybe you can implement like below code in two places ( run and saveMetaData_currentWeather methods)

Oh yes, this is actually the problem. It's not the updateProgress, but this line above in CurrentWeatherWorker::run():

 emit log("data object read");

Because reading the data is done really fast and about 170 000 calls in about 10 seconds seems too be to much to handle for the mainwindow (the message gets printed every time). The queue of the received signals is just too long.
Anyway, changing the updateProgress to something like this:

        if(i%(metaList.length()/100)==0)
            emit updateProgress(qRound(100.0*(qreal)i/(qreal)metaList.length()));

should increase performance sagnificantly while the result should stay the same.

@Christian-Ehrlicher said in QThread with SQLite blocks GUI:

This is wrong when you're more than one db connection - you want 'QSqlQuery query(db)'

Yes, thanks. So far it even works the old way since I am only working on one database at a time yet, but I plan to run multiple workers simultaneously.

Thank you!!