How to create a QTreeView that can handle very large amount of data?

Daniella

I'm adding data loaded form a file on disk to a QTreeView , the file contains more than 60.000 rows and new 'rows'/data will be constantly added to it while its being executed, so each time the GUI is reloaded/reopened the file will contain more data.

I tried to load this data using another thread just to figure out that QAbstractItemModel is not thread safe.

Then I tried to created a 'lazy load' subclass of a QAbstractItemModel , that 'process' the items as they get visible in the
QTreeview.
At the beginning it worked great, as it doesn't freeze the entire GUI for like 10~ seconds at runtime anymore.

However when i scroll the QTreeView it get frozen for many seconds, i think its 'loading' the data.

I was reading the documentation of QAbstractItemModel and see that it have these two functions: fetchMore() and canFetchMore(), to control how the data is loaded, but i didn't understand how to use it and if they could help in this case.

How I could dynamically load/unload the data as it get visible in the QTreeView? and free the memory of the rows that are not visible/not visible anymore, instead of having the entire file loaded into the memory.

Here is what i already did, a reproducible example:

#include "treeview.h"

MainWindow::MainWindow(QWidget *parent)
    : QMainWindow(parent)
    , ui(new Ui::MainWindowClass())
{
    ui->setupUi(this);

    QGridLayout* layout = new QGridLayout(this);
    ui->centralWidget->setLayout(layout);
    auto treeView = new TreeView(this);
    layout->addWidget(treeView);
  
    QShortcut *shortcut = new QShortcut(QKeySequence(Qt::Key_F2), this);
    connect(shortcut, &QShortcut::activated, [=] {
        treeView->setInfo();
        qDebug() << "rowCount: " << treeView->model->rowCount();
    });
}

treeview.h

class LazyLoadingModel : public QAbstractItemModel
{
    Q_OBJECT

public:
    LazyLoadingModel(QObject *parent = nullptr) : QAbstractItemModel(parent) {}

    int rowCount(const QModelIndex &parent = QModelIndex()) const override
    {
        return m_rows.count();
    }

    int columnCount(const QModelIndex &parent = QModelIndex()) const override
    {
        return m_columns.count();
    }

    QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const override
    {
        if (!index.isValid())
            return QVariant();

        int row = index.row();
        int column = index.column();

        if (row >= m_rows.count() || column >= m_columns.count())
            return QVariant();

        if (role == Qt::DisplayRole)
        {
            return m_rows[row][column];
        }

        return QVariant();
    }

    bool setData(const QModelIndex &index, const QVariant &value, int role = Qt::EditRole) override
    {
        if (!index.isValid())
            return false;

        int row = index.row();
        int column = index.column();

        if (row >= m_rows.count() || column >= m_columns.count())
            return false;

        if (role == Qt::EditRole)
        {
            m_rows[row][column] = value.toString();
            emit dataChanged(index, index);
            return true;
        }

        return false;
    }

    QModelIndex index(int row, int column, const QModelIndex &parent = QModelIndex()) const override
    {
        if (!parent.isValid())
            return createIndex(row, column);
        else
            return QModelIndex();
    }

    QModelIndex parent(const QModelIndex &index) const override
    {
        return QModelIndex();
    }

    void setColumns(const QStringList &columns)
    {
        m_columns = columns;
    }

    void addData(const QVector<QStringList> &row_info)
    {
        beginInsertRows(QModelIndex(), m_rows.count(), m_rows.count() + row_info.count() - 1);
        for (const auto &row : row_info)
            m_rows.append(row);
        endInsertRows();
    }

private:
    QStringList m_columns;
    QList<QStringList> m_rows;
};



class TreeView : public QTreeView {
    Q_OBJECT
public:
    LazyLoadingModel* model = new LazyLoadingModel();

    TreeView(QWidget* parent = 0) : QTreeView(parent) {
        setModel(model);
    }

    void setInfo() {
        QElapsedTimer timer;
        timer.start();

        model->setColumns({ "Column 0", "Column 1", "Column 2", "Column 3" });

        //
        // Load the data from disk and parse it into a qvector
        // or:
        // Create some random data, just to debug the model:
        QVector<QStringList> info{};
        QStringList list;
        for (size_t i = 0; i < 60000; i++) {
            list = {};
            QString str = QString::number(i);

            for (size_t i = 0; i < 4; i++)
                list << str;

            info.emplace_back(list);
        }

        model->addData(info);

        qint64 elapsed = timer.elapsed();
        qDebug() << "[T] Elapsed time (ms):" << elapsed;
    }

};

Christian Ehrlicher

Move the parsing stuff into own thread and set the data at once into your model (in the main thread) afterwards.

jeremy_k

@Daniella said in How to create a QTreeView that can handle very large amount of data?:

I'm adding data loaded form a file on disk to a QTreeView , the file contains more than 60.000 rows and new 'rows'/data will be constantly added to it while its being executed, so each time the GUI is reloaded/reopened the file will contain more data.

Is 60.000 records a lot of data to hold in memory? 60k integers is probably not a big deal for a currently produced desktop or smartphone level device. 60k 4K HDR movies is a different story.

I tried to load this data using another thread just to figure out that QAbstractItemModel is not thread safe.

Then I tried to created a 'lazy load' subclass of a QAbstractItemModel , that 'process' the items as they get visible in the
QTreeview.
At the beginning it worked great, as it doesn't freeze the entire GUI for like 10~ seconds at runtime anymore.

However when i scroll the QTreeView it get frozen for many seconds, i think its 'loading' the data.

Profile the execution. Otherwise you're likely engaging (and asking us to engage) in premature optimization.

I was reading the documentation of QAbstractItemModel and see that it have these two functions: fetchMore() and canFetchMore(), to control how the data is loaded, but i didn't understand how to use it and if they could help in this case.

The view calls these in situations where more data might be useful. Look at them as an opportunity to queue asynchronous loading.

if (view.scrollBar.value() == view.scrollBar.maximum())
    if (canFetchMore())
        fetchMore();
...

struct Model: public QAbstractItemModel {
  int pageCount;
  int pagesLoaded;
  QNetworkManager netman;

 Model() {
   connect(&netman, &QNetworkManager::finished, this, &Model::addData);
 }

  bool canFetchMore(QModelIndex &parent) {
    return pageCount > pagesLoaded;
  }

  void fetchMore(QModelIndex &parent) {
     if (pageCount > pagesLoaded)
         netman.get(QUrl(...));
  }
};

How I could dynamically load/unload the data as it get visible in the QTreeView? and free the memory of the rows that are not visible/not visible anymore, instead of having the entire file loaded into the memory.

QAbstractItemView::indexAt() can help answer this question, but be careful. If the user scrolls and the view attempts to display what it thinks is available data, the model will need to either block in QAbstractItemModel::data() while retrieving it, or present dummy data to later be corrected with QAbstractItemMode::dataChanged(). Is unloading necessary?

    void addData(const QVector<QStringList> &row_info)
    {
        beginInsertRows(QModelIndex(), m_rows.count(), m_rows.count() + row_info.count() - 1);
        for (const auto &row : row_info)
            m_rows.append(row);
        endInsertRows();
    }

Engaging in a little premature optimization, consider QList::append(QList&&) or QList::append::(QList &) instead of a for loop picking apart the list.