Large set of data in QTableView

FroZtiZ

I am trying to add large sets of data in QTableView. As there is huge set of data, in order to avoid freezing the application, I am trying to use a QAbstractTableModel to not treat the whole dataset immediately, but only what is necessary. Based on the QT example for this (HERE), I have the following class:
FileListModel.cpp

#include "filelistmodel.h"

#include <QGuiApplication>
#include <QDir>
#include <QPalette>
#include "qdebug.h"

FileListModel::FileListModel(QObject *parent)
    : QAbstractTableModel(parent), fileCount(0) //edit
{}

int FileListModel::rowCount(const QModelIndex &parent) const
{
    return parent.isValid() ? 0 : fileCount;
}

QVariant FileListModel::data(const QModelIndex &index, int role) const
{
    if (!index.isValid())
    {
        return QVariant();
    }
    if (index.row() >= fileList.size() || index.row() < 0)
    {
        return QVariant();
    }
    if (role == Qt::DisplayRole)
    {
        return fileList.at(index.row());
    }
    return QVariant();
}

bool FileListModel::canFetchMore(const QModelIndex &parent) const
{
    if (parent.isValid())
    {
        return false;
    }
    return (fileCount < fileList.size());
}
int FileListModel::columnCount(const QModelIndex &parent) const
{
    return parent.isValid() ? 0 : colCount;
}
void FileListModel::fetchMore(const QModelIndex &parent)
{
    if (parent.isValid())
    {
        return;
    }
    int remainder = fileList.size() - fileCount;
    int itemsToFetch = qMin(100, remainder);

    if (itemsToFetch <= 0)
    {
        return;
    }
    beginInsertRows(QModelIndex(), fileCount, fileCount + itemsToFetch - 1);
    qDebug()<< "Qmodelindex "<< QModelIndex()<< "filecount "<< fileCount <<"filecount + itemtofetch "<<fileCount + itemsToFetch - 1;
    fileCount += itemsToFetch;

    endInsertRows();
}
void FileListModel::setColumnNumber(const int x) //edit
{
    colCount = x;
}
void FileListModel::setDataToList(const QList<float> &data)
{
    beginResetModel();
    fileList = data;
    fileCount = 0;
    endResetModel();
}

FileListModel.h

#ifndef FILELISTMODEL_H
#define FILELISTMODEL_H

#include <QAbstractTableModel>
#include <QStringList>

class FileListModel : public QAbstractTableModel //edit
{
    Q_OBJECT

public:
    FileListModel(QObject *parent = nullptr);

    int rowCount(const QModelIndex &parent = QModelIndex()) const override;
    QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const override;
    int columnCount(const QModelIndex &parent = QModelIndex()) const override; //edit
    void setColumnNumber(const int );

public slots:
    void setDataToList(const QList<float>&);

protected:
    bool canFetchMore(const QModelIndex &parent) const override;
    void fetchMore(const QModelIndex &parent) override;

private:
    QList<float> fileList;
    int fileCount;
int colCount;//edit
};

#endif // FILELISTMODEL_H

In my window, I have a QTableView and 5 QList<float>, each representing a column. I proceed this way to add the data in my QTableView:

FileListModel *model=new FileListModel(this);
model->setColumnNumber(5); //edit

model->setDataToList(list1);
model->setDataToList(list2);
model->setDataToList(list3);
model->setDataToList(list4);
model->setDataToList(list5);

tableview->setModel(model)

However, the last list added (list5) replace the previous one and I have only one column. I am aware that I need to write the needed code to add columns. But to be honest, I am not knowing very well the QAbstract classes, and have no idea how to proceed. I would be grateful if you could provide me some hints or an example on how to modify my code in order to add columns in my model.

JonB

@FroZtiZ said in Large set of data in QTreeView:

I am trying to add large sets of data in QTreeView. [...] I am trying to use a QAbstractListModel

I am aware that I need to write the needed code to add columns

What is this about? What is the whole question about "columns"? You are using a QAbstractListModel, this is for just 1 column. And then https://doc.qt.io/qt-5/qabstractlistmodel.html#details:

QAbstractListModel provides a standard interface for models that represent their data as a simple non-hierarchical sequence of items

Since the model provides a more specialized interface than QAbstractItemModel, it is not suitable for use with tree views; you will need to subclass QAbstractItemModel if you want to provide a model for that purpose. If you need to use a number of list models to manage data, it may be more appropriate to subclass QAbstractTableModel instead.

Separately, when you start having a FileList class and a QTreeViiew, are you at least aware of https://doc.qt.io/qt-5/search-results.html?q=qfilesystemmodel, and say https://www.bogotobogo.com/Qt/Qt5_QTreeView_QFileSystemModel_ModelView_MVC.php ? Or, is your FileListModel with QList<float> fileList nothing to do with files?

FroZtiZ

Ok, thanks, that is an interesting remark. In this case I cannot use a QAbstractListModel. I will reformulate my question:
I want to add a huge set of data in a table. I choose QTreeView for the table as, I prefer the format (small items) compared to QTableView. I have no idea which one is the most efficient.
My goal is only to show the data in a QTreeView. There are 5 columns and 20 000 rows. If I simply use QStandardItem, the construction and destruction of the table is very slow. Hence, from the fetch example in QT website, I am trying to only show the needed data in my table. How can I succeed to do this. Do you have an example I could adapt?

FroZtiZ

Thanks to your reply, I performed some changes in the code and I know succeed to add 5 columns. Unfortunately, the 5 columns are the same and I will try to change this

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTreeView:

I prefer the format (small items) compared to QTableView

That's no reason to use a QTreeView since you can set the height by your own. If you care for performance then use the view which matches your model and don't mis-use another view.

FroZtiZ

Thanks Christian, I changed the QTreeView to QTableView. Do you know how I can add a column in my model that will follow the same conditions that i am looking for? I mean, the other columns (like the first) should show row only when the user scroll down in the table, to avoid loading too many data simultaneously

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

should show row only when the user scroll down in the table, to avoid loading too many data simultaneously

How large is your dataset so that you think this is neede? See QAIV::canFetchMore()

FroZtiZ

Minimum 5 columns of 20 000 rows. It s needed because I tried already and that is very very slow. I tried with QTableWidget, QTreeWidget.

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

I tried already and that is very very slow

Create a proper model, use a profiler to see where the time is spent. 20k items is not that larger that lazy loading is needed I would say.

FroZtiZ

Thanks for your remark. I have already used ways to see where the time is spent (at the construction and destruction). And it corresponds to 100k items at the minimum. Regarding the proper model, I am trying, with the code lines provided in my first post above. My question is now: How can I add the different columns to the table by providing QList<float> without losing the Fetch methods. My question is not to know if it is necessary to create a model as I have done many tests previously.

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

(at the construction and destruction)

Which construction? Creating 50k of floats? I don't think the model is the problem here...

How can I add the different columns to the table by providing QList<float> without losing the Fetch methods.

Please be more specific - what are you trying to display - only floats? and why a list? isn't an array/vector the better container?). Your code only shows one column - is this true for your real application? If so how is your data organized?

FroZtiZ

I will try to be more specific.

The construction and destruction of QStandardItem in a QTableWidget is very slow. That is the reason why I decided to construct a QAbstractTableModel. The construction and destruction of the QList is very fast and it is not the problem.

The current model that is described in my code above is very fast so, it seems to be the solution. Inside, I want to put 5 QList, each being a column of this model. The objective is to put this model in a table (column, row). The 5 QList have each, minimum 20 000 rows. I succeed to create a QAbstractTableModel with 5 columns that only show the 100 first rows, and when we scroll down, it shows the 100 next etc...
This process is very efficient (understand fast) however, I have a coding difficulty.

The model I wrote in my first message above only takes as input one QList that he displays in the N columns asked.
I want this model to take as input N QList displayed each in the corresponding N columns. And an additional condition is that when the user scroll down, the rows are showed little by little for the N columns as it is currently the case for my single column.

Could you please help me to improve my class in order to have a method "model->addColumn(int i, QList<float> x)" where int i is the columnIndex and QList<float> x is the QList I want to set in the column i?

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

QList

again: you have a array/vector, not list! So use a QVector..

The current model that is described in my code above is very fast so

fine, but why you need the fetch stuff then?

FroZtiZ

Thank you for your contributions. I found how to proceed by storing my data in a QList<QList<float>>.
Please find below a working code that can probably be improved:

CPP FILE


#include "filelistmodel.h"

#include <QGuiApplication>
#include <QDir>
#include <QPalette>
#include "qdebug.h"

FileListModel::FileListModel(QObject *parent)
    : QAbstractTableModel(parent), rowNumber(0)
{}

int FileListModel::rowCount(const QModelIndex &parent) const
{
    return parent.isValid() ? 0 : rowNumber;
}

int FileListModel::columnCount(const QModelIndex &parent) const
{
    return parent.isValid() ? 0 : colNumber;
}

void FileListModel::setColumnNumber(const int x)
{
    colNumber = x;
}

QVariant FileListModel::data(const QModelIndex &index, int role) const
{
    if (!index.isValid())
    {
        return QVariant();
    }
    if (index.row() >= fileList[0].size() || index.row() < 0)
    {
        return QVariant();
    }
    if (role == Qt::DisplayRole)
    {
        return fileList[index.column()][index.row()];
    }
    return QVariant();
}

bool FileListModel::canFetchMore(const QModelIndex &parent) const
{
    if (parent.isValid())
    {
        return false;
    }
    return (rowNumber < fileList[0].size());
}

void FileListModel::fetchMore(const QModelIndex &parent)
{
    if (parent.isValid())
    {
        return;
    }
    int remainder = fileList[0].size() - rowNumber;
    int itemsToFetch = qMin(100, remainder);

    if (itemsToFetch <= 0)
    {
        return;
    }
    beginInsertRows(QModelIndex(), rowNumber, rowNumber + itemsToFetch - 1);

    rowNumber += itemsToFetch;

    endInsertRows();
}

void FileListModel::setDataToTable(const QList<QList<float>> &data)
{
    beginResetModel();
    fileList = data;
    rowNumber = 0;
    endResetModel();
}

H FILE


#ifndef FILELISTMODEL_H
#define FILELISTMODEL_H

#include <QAbstractListModel>
#include <QStringList>

class FileListModel : public QAbstractTableModel
{
    Q_OBJECT

public:
    FileListModel(QObject *parent = nullptr);

    int rowCount(const QModelIndex &parent = QModelIndex()) const override;
    int columnCount(const QModelIndex &parent = QModelIndex()) const override;
    QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const override;
    void setColumnNumber(const int );
    void setDataToTable(const QList<QList<float>>&);

protected:
    bool canFetchMore(const QModelIndex &parent) const override;
    void fetchMore(const QModelIndex &parent) override;

private:
    QList<QList<float>> fileList;
    int rowNumber;
    int colNumber;
};

#endif // FILELISTMODEL_H

BUILDING MODEL

FileListModel *pModel =new FileListModel(this);
QList<QList<float>> a;
pModel->setColumnNumber(5);
a.append(qlist1);
a.append(qlist2);
a.append(qlist3);
a.append(qlist4);
a.append(qlist5);
pModel->setDataToTable(a);

Christian Ehrlicher

You're still using the wrong container. You want to use a vector, not a list. Your fetchMore() is trivial - I doubt this is your real workcase, or?

FroZtiZ

Why should I use a QVector instead of a QList?
My fetch is maybe trivial but it is what I was trying to do. I am not a QT expert, especially, I dont know at all the abstract classes.

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

Why should I use a QVector instead of a QList?

Because QList is a list (more or less) and QVector a vector. See also the QList documentation - since you're using floats, the memory needed for QList<float> is exactly double the memory needed for QVector<float> on 64 bit systems.

FroZtiZ

Thanks I will perform the modification

JonB

@FroZtiZ
Excuse my chiming in again. I have to confess I am lost as to whether you are now asking about which model to use, columns vs rows, efficiency of adding, or ...

@FroZtiZ about 18 hours ago

Minimum 5 columns of 20 000 rows

@Christian-Ehrlicher
20k items is not that larger that lazy loading is needed I would say

First, always heed what @Christian-Ehrlicher asks/suggests, as he knows more than I. However, I am slightly surprised, I would have said 20k (x 5 columns) is quite large. I don't know what your "very very slow" actually means. But my thought would be: if you are presenting a UI like a QTableWidget or whatever, why would you want 20k items in it? The user cannot handle that many items to scroll through....

FroZtiZ

I am doing what Christian suggested. And now, with the last modifications, everything is working. So I am not asking anymore about how to proceed with columns, rows or model.
When I say very very slow, I mean I need ~5 seconds to update/construct/destroy the table.
I agree that 100k items is quite large for a user to read everything. In fact, these items are spread in different tables that represent x,y coordinates of a function. For sure the user won't read everything simultaneously. However, I need to be able to provide the data on request inside the user interface.