Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.

parameter2

Hello. I have a sortable QTableView that is used to display a log from a CSV file.

Here is how I initialize the QTableView:

stdmodel_test = new QStandardItemModel(0,4,this);
stdmodel_test->setHorizontalHeaderItem(0, new QStandardItem(QString("Column1")));
stdmodel_test->setHorizontalHeaderItem(1, new QStandardItem(QString("Column2")));
stdmodel_test->setHorizontalHeaderItem(2, new QStandardItem(QString("Column3")));

sort_test = new QSortFilterProxyModel(this);
sort_test->setDynamicSortFilter(true);
sort_test->setSourceModel(stdmodel_test);

ui->tableView_test->setModel(sort_test);
ui->tableView_test->setEditTriggers(QAbstractItemView::NoEditTriggers);
ui->tableView_test->setFocusPolicy(Qt::NoFocus);
ui->tableView_test->setColumnWidth(0,150);
ui->tableView_test->setColumnWidth(1,150);
ui->tableView_test->setColumnWidth(2,150);

ui->tableView_test->setVerticalScrollBarPolicy(Qt::ScrollBarAlwaysOn);
ui->tableView_test->setHorizontalScrollBarPolicy(Qt::ScrollBarAlwaysOff);
ui->tableView_test->verticalHeader()->setVisible(false);
ui->tableView_test->setSortingEnabled(true);
ui->tableView_test->sortByColumn(0,Qt::AscendingOrder);

Here is how I load (and reload) the data into the view:

void MainWindow::load_testLog() {
stdmodel_test->setRowCount(0);

tmpmodel_test = new QStandardItemModel(0,4);
tmpmodel_test->setHorizontalHeaderItem(0, new QStandardItem(QString("Column1")));
tmpmodel_test->setHorizontalHeaderItem(1, new QStandardItem(QString("Column2")));
tmpmodel_test->setHorizontalHeaderItem(2, new QStandardItem(QString("Column3")));

tmpmodel_test->setRowCount(0);

QFile file;
file.setFileName(test.csv);

if (file.open(QIODevice::ReadOnly)) {
    int lineindex = 0;                     // file line counter
    QTextStream in(&file);                 // read to text stream

    while (!in.atEnd()) {

        QString fileLine = in.readLine();
        QStringList lineToken = fileLine.split(",", QString::KeepEmptyParts);
        for (int j = 0; j < 3; j++) {
            const QString& value = lineToken.at(j);

            QStandardItem *item = new QStandardItem(value);
            tmpmodel_test->setItem(lineindex, j, item);
        }
        lineindex++;
    }
    file.close();

    sort_test->setSourceModel(tmpmodel_test);

    ui->tableView_test->setModel(sort_test);
}
}

And finally this is how call the load_testLog function:

void MainWindow::reload(){

QFuture<void> t0 = QtConcurrent::run(this,&MainWindow::load_testLog);
}

Here is a sample log.csv file with a few lines:

00-00-0000 12:00:10.001,TEST LOG ENTRY,System (TEST_LOG)
00-00-0000 12:00:10.002,TEST LOG ENTRY,System (TEST_LOG)
00-00-0000 12:00:10.003,TEST LOG ENTRY,System (TEST_LOG)

Even though my approach sort of works, and the log seems to load properly into QTableview, I get a bunch of worrying messagess in the application output:

QObject: Cannot create children for a parent that is in a different thread.
(Parent is QHeaderView(0x564279395eb0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
QBasicTimer::start: Timers cannot be started from another thread
QObject: Cannot create children for a parent that is in a different thread.
(Parent is QHeaderView(0x5642793637b0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
QBasicTimer::start: Timers cannot be started from another thread
QObject: Cannot create children for a parent that is in a different thread.
(Parent is QTableView(0x56427938f9a0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)

Would you help me fix this? Fyi. I need to load all the data into the log at once.

Asperamanca

The part where you set the model inside the thread is the core problem. As mentioned, there are multiple solutions.

Christian is right, QtConcurrent::map it's not really needed here. I think the easiest solution is to use a QFutureWatcher and connect to the "finished" signal. In the called slot, you can set the model (in context of the UI thread).

The only question is: How to get access to the result?
Multiple options:

QtConcurrent::map would put the result in the QFuture (although this isn't really the use case for QtConcurrent::map)
You can pass something to the QtConcurrent::run function where it can put the result. Of course, in this case, proper locking is your job
You can, of course, use QThread instead, but unless you plan to do more than that inside the thread, I see no compelling reason to do so

Asperamanca

@parameter2 said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

ui->tableView_events->setModel(sort_test);

You can't just call the tableView from a different thread like this.
I would suggest that instead of using QtConcurrent::run, you use QtConcurrent::map and map from an input file to a complete QAbstractItemModel.

That way, you can get the final model from the QFuture once the model is complete, and do the setModel inside the UI thread.

parameter2

@Asperamanca said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

I would suggest that instead of using QtConcurrent::run, you use QtConcurrent::map and map from an input file to a complete QAbstractItemModel.

Thank you for your suggestions. Is there any way you could show me how to accomplish that with QtConcurrent::map using some minimal example?

Christian Ehrlicher

No need for QtConcurrent::map(). A separate thread or even better QtConcurrent::run() is fine here.

parameter2

@Christian-Ehrlicher said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

No need for QtConcurrent::map(). A separate thread or even better QtConcurrent::run() is fine here.

What might I be doing wrong, since I'm using QtConcurrent::run() to populate the model and getting all these QThread errors?

QObject: Cannot create children for a parent that is in a different thread.
(Parent is QHeaderView(0x564279395eb0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
QBasicTimer::start: Timers cannot be started from another thread
QObject: Cannot create children for a parent that is in a different thread.
(Parent is QHeaderView(0x5642793637b0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
QBasicTimer::start: Timers cannot be started from another thread
QObject: Cannot create children for a parent that is in a different thread.
(Parent is QTableView(0x56427938f9a0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)

Christian Ehrlicher

@parameter2 @Asperamanca already told you what you're doing wrong and how to fix it.

Asperamanca

The part where you set the model inside the thread is the core problem. As mentioned, there are multiple solutions.

Christian is right, QtConcurrent::map it's not really needed here. I think the easiest solution is to use a QFutureWatcher and connect to the "finished" signal. In the called slot, you can set the model (in context of the UI thread).

The only question is: How to get access to the result?
Multiple options:

QtConcurrent::map would put the result in the QFuture (although this isn't really the use case for QtConcurrent::map)
You can pass something to the QtConcurrent::run function where it can put the result. Of course, in this case, proper locking is your job
You can, of course, use QThread instead, but unless you plan to do more than that inside the thread, I see no compelling reason to do so

Christian Ehrlicher

Use QtConcurrent::run() and get the result through the future.

parameter2

@Asperamanca said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

The part where you set the model inside the thread is the core problem. As mentioned, there are multiple solutions.

Christian is right, QtConcurrent::map it's not really needed here. I think the easiest solution is to use a QFutureWatcher and connect to the "finished" signal. In the called slot, you can set the model (in context of the UI thread).

The only question is: How to get access to the result?
Multiple options:

QtConcurrent::map would put the result in the QFuture (although this isn't really the use case for QtConcurrent::map)

You can pass something to the QtConcurrent::run function where it can put the result. Of course, in this case, proper locking is your job

You can, of course, use QThread instead, but unless you plan to do more than that inside the thread, I see no compelling reason to do so

@Christian-Ehrlicher said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

Use QtConcurrent::run() and get the result through the future.

Thank you. I appreciate your responses. Hopefully I can figure this out now.

Asperamanca

@Christian-Ehrlicher said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

Use QtConcurrent::run() and get the result through the future.

I didn't know you can do that! I though for QtConcurrent::run the future object is void.
Shows you learn new tricks every day :-)

Pl45m4

@Asperamanca

If you set your QFuture's type to a type T != void, you actually can "use" the result later. Of course it has to match the return type of the function used in run.

As described here