Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.



  • Hello. I have a sortable QTableView that is used to display a log from a CSV file.

    Here is how I initialize the QTableView:

    stdmodel_test = new QStandardItemModel(0,4,this);
    stdmodel_test->setHorizontalHeaderItem(0, new QStandardItem(QString("Column1")));
    stdmodel_test->setHorizontalHeaderItem(1, new QStandardItem(QString("Column2")));
    stdmodel_test->setHorizontalHeaderItem(2, new QStandardItem(QString("Column3")));
    
    sort_test = new QSortFilterProxyModel(this);
    sort_test->setDynamicSortFilter(true);
    sort_test->setSourceModel(stdmodel_test);
    
    ui->tableView_test->setModel(sort_test);
    ui->tableView_test->setEditTriggers(QAbstractItemView::NoEditTriggers);
    ui->tableView_test->setFocusPolicy(Qt::NoFocus);
    ui->tableView_test->setColumnWidth(0,150);
    ui->tableView_test->setColumnWidth(1,150);
    ui->tableView_test->setColumnWidth(2,150);
    
    ui->tableView_test->setVerticalScrollBarPolicy(Qt::ScrollBarAlwaysOn);
    ui->tableView_test->setHorizontalScrollBarPolicy(Qt::ScrollBarAlwaysOff);
    ui->tableView_test->verticalHeader()->setVisible(false);
    ui->tableView_test->setSortingEnabled(true);
    ui->tableView_test->sortByColumn(0,Qt::AscendingOrder);
    

    Here is how I load (and reload) the data into the view:

    void MainWindow::load_testLog() {
    stdmodel_test->setRowCount(0);
    
    tmpmodel_test = new QStandardItemModel(0,4);
    tmpmodel_test->setHorizontalHeaderItem(0, new QStandardItem(QString("Column1")));
    tmpmodel_test->setHorizontalHeaderItem(1, new QStandardItem(QString("Column2")));
    tmpmodel_test->setHorizontalHeaderItem(2, new QStandardItem(QString("Column3")));
    
    tmpmodel_test->setRowCount(0);
    
    QFile file;
    file.setFileName(test.csv);
    
    if (file.open(QIODevice::ReadOnly)) {
        int lineindex = 0;                     // file line counter
        QTextStream in(&file);                 // read to text stream
    
        while (!in.atEnd()) {
    
            QString fileLine = in.readLine();
            QStringList lineToken = fileLine.split(",", QString::KeepEmptyParts);
            for (int j = 0; j < 3; j++) {
                const QString& value = lineToken.at(j);
    
                QStandardItem *item = new QStandardItem(value);
                tmpmodel_test->setItem(lineindex, j, item);
            }
            lineindex++;
        }
        file.close();
    
        sort_test->setSourceModel(tmpmodel_test);
    
        ui->tableView_test->setModel(sort_test);
    }
    }
    

    And finally this is how call the load_testLog function:

    void MainWindow::reload(){
    
    QFuture<void> t0 = QtConcurrent::run(this,&MainWindow::load_testLog);
    }
    

    Here is a sample log.csv file with a few lines:

    00-00-0000 12:00:10.001,TEST LOG ENTRY,System (TEST_LOG)
    00-00-0000 12:00:10.002,TEST LOG ENTRY,System (TEST_LOG)
    00-00-0000 12:00:10.003,TEST LOG ENTRY,System (TEST_LOG)
    

    Even though my approach sort of works, and the log seems to load properly into QTableview, I get a bunch of worrying messagess in the application output:

    QObject: Cannot create children for a parent that is in a different thread.
    (Parent is QHeaderView(0x564279395eb0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
    QBasicTimer::start: Timers cannot be started from another thread
    QObject: Cannot create children for a parent that is in a different thread.
    (Parent is QHeaderView(0x5642793637b0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
    QBasicTimer::start: Timers cannot be started from another thread
    QObject: Cannot create children for a parent that is in a different thread.
    (Parent is QTableView(0x56427938f9a0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
    

    Would you help me fix this? Fyi. I need to load all the data into the log at once.



  • The part where you set the model inside the thread is the core problem. As mentioned, there are multiple solutions.

    Christian is right, QtConcurrent::map it's not really needed here. I think the easiest solution is to use a QFutureWatcher and connect to the "finished" signal. In the called slot, you can set the model (in context of the UI thread).

    The only question is: How to get access to the result?
    Multiple options:

    • QtConcurrent::map would put the result in the QFuture (although this isn't really the use case for QtConcurrent::map)
    • You can pass something to the QtConcurrent::run function where it can put the result. Of course, in this case, proper locking is your job
    • You can, of course, use QThread instead, but unless you plan to do more than that inside the thread, I see no compelling reason to do so


  • @parameter2 said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

    ui->tableView_events->setModel(sort_test);

    You can't just call the tableView from a different thread like this.
    I would suggest that instead of using QtConcurrent::run, you use QtConcurrent::map and map from an input file to a complete QAbstractItemModel.

    That way, you can get the final model from the QFuture once the model is complete, and do the setModel inside the UI thread.



  • @Asperamanca said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

    I would suggest that instead of using QtConcurrent::run, you use QtConcurrent::map and map from an input file to a complete QAbstractItemModel.

    Thank you for your suggestions. Is there any way you could show me how to accomplish that with QtConcurrent::map using some minimal example?


  • Lifetime Qt Champion

    No need for QtConcurrent::map(). A separate thread or even better QtConcurrent::run() is fine here.



  • @Christian-Ehrlicher said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

    No need for QtConcurrent::map(). A separate thread or even better QtConcurrent::run() is fine here.

    What might I be doing wrong, since I'm using QtConcurrent::run() to populate the model and getting all these QThread errors?

    QObject: Cannot create children for a parent that is in a different thread.
    (Parent is QHeaderView(0x564279395eb0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
    QBasicTimer::start: Timers cannot be started from another thread
    QObject: Cannot create children for a parent that is in a different thread.
    (Parent is QHeaderView(0x5642793637b0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
    QBasicTimer::start: Timers cannot be started from another thread
    QObject: Cannot create children for a parent that is in a different thread.
    (Parent is QTableView(0x56427938f9a0), parent's thread is QThread(0x564278e9b540), current thread is QThread(0x56427974f4b0)
    

  • Lifetime Qt Champion

    @parameter2 @Asperamanca already told you what you're doing wrong and how to fix it.



  • The part where you set the model inside the thread is the core problem. As mentioned, there are multiple solutions.

    Christian is right, QtConcurrent::map it's not really needed here. I think the easiest solution is to use a QFutureWatcher and connect to the "finished" signal. In the called slot, you can set the model (in context of the UI thread).

    The only question is: How to get access to the result?
    Multiple options:

    • QtConcurrent::map would put the result in the QFuture (although this isn't really the use case for QtConcurrent::map)
    • You can pass something to the QtConcurrent::run function where it can put the result. Of course, in this case, proper locking is your job
    • You can, of course, use QThread instead, but unless you plan to do more than that inside the thread, I see no compelling reason to do so

  • Lifetime Qt Champion

    • Use QtConcurrent::run() and get the result through the future.


  • @Asperamanca said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

    The part where you set the model inside the thread is the core problem. As mentioned, there are multiple solutions.

    Christian is right, QtConcurrent::map it's not really needed here. I think the easiest solution is to use a QFutureWatcher and connect to the "finished" signal. In the called slot, you can set the model (in context of the UI thread).

    The only question is: How to get access to the result?
    Multiple options:

    • QtConcurrent::map would put the result in the QFuture (although this isn't really the use case for QtConcurrent::map)
    • You can pass something to the QtConcurrent::run function where it can put the result. Of course, in this case, proper locking is your job
    • You can, of course, use QThread instead, but unless you plan to do more than that inside the thread, I see no compelling reason to do so

    @Christian-Ehrlicher said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

    • Use QtConcurrent::run() and get the result through the future.

    Thank you. I appreciate your responses. Hopefully I can figure this out now.



  • @Christian-Ehrlicher said in Best way to efficiently load 20MB of data from CSV file into QTableView without hanging the gui thread.:

    • Use QtConcurrent::run() and get the result through the future.

    I didn't know you can do that! I though for QtConcurrent::run the future object is void.
    Shows you learn new tricks every day :-)



  • @Asperamanca

    If you set your QFuture's type to a type T != void, you actually can "use" the result later. Of course it has to match the return type of the function used in run.

    As described here


Log in to reply