Large set of data in QTableView

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

should show row only when the user scroll down in the table, to avoid loading too many data simultaneously

How large is your dataset so that you think this is neede? See QAIV::canFetchMore()

FroZtiZ

Minimum 5 columns of 20 000 rows. It s needed because I tried already and that is very very slow. I tried with QTableWidget, QTreeWidget.

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

I tried already and that is very very slow

Create a proper model, use a profiler to see where the time is spent. 20k items is not that larger that lazy loading is needed I would say.

FroZtiZ

Thanks for your remark. I have already used ways to see where the time is spent (at the construction and destruction). And it corresponds to 100k items at the minimum. Regarding the proper model, I am trying, with the code lines provided in my first post above. My question is now: How can I add the different columns to the table by providing QList<float> without losing the Fetch methods. My question is not to know if it is necessary to create a model as I have done many tests previously.

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

(at the construction and destruction)

Which construction? Creating 50k of floats? I don't think the model is the problem here...

How can I add the different columns to the table by providing QList<float> without losing the Fetch methods.

Please be more specific - what are you trying to display - only floats? and why a list? isn't an array/vector the better container?). Your code only shows one column - is this true for your real application? If so how is your data organized?

FroZtiZ

I will try to be more specific.

The construction and destruction of QStandardItem in a QTableWidget is very slow. That is the reason why I decided to construct a QAbstractTableModel. The construction and destruction of the QList is very fast and it is not the problem.

The current model that is described in my code above is very fast so, it seems to be the solution. Inside, I want to put 5 QList, each being a column of this model. The objective is to put this model in a table (column, row). The 5 QList have each, minimum 20 000 rows. I succeed to create a QAbstractTableModel with 5 columns that only show the 100 first rows, and when we scroll down, it shows the 100 next etc...
This process is very efficient (understand fast) however, I have a coding difficulty.

The model I wrote in my first message above only takes as input one QList that he displays in the N columns asked.
I want this model to take as input N QList displayed each in the corresponding N columns. And an additional condition is that when the user scroll down, the rows are showed little by little for the N columns as it is currently the case for my single column.

Could you please help me to improve my class in order to have a method "model->addColumn(int i, QList<float> x)" where int i is the columnIndex and QList<float> x is the QList I want to set in the column i?

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

QList

again: you have a array/vector, not list! So use a QVector..

The current model that is described in my code above is very fast so

fine, but why you need the fetch stuff then?

FroZtiZ

Thank you for your contributions. I found how to proceed by storing my data in a QList<QList<float>>.
Please find below a working code that can probably be improved:

CPP FILE


#include "filelistmodel.h"

#include <QGuiApplication>
#include <QDir>
#include <QPalette>
#include "qdebug.h"

FileListModel::FileListModel(QObject *parent)
    : QAbstractTableModel(parent), rowNumber(0)
{}

int FileListModel::rowCount(const QModelIndex &parent) const
{
    return parent.isValid() ? 0 : rowNumber;
}

int FileListModel::columnCount(const QModelIndex &parent) const
{
    return parent.isValid() ? 0 : colNumber;
}

void FileListModel::setColumnNumber(const int x)
{
    colNumber = x;
}

QVariant FileListModel::data(const QModelIndex &index, int role) const
{
    if (!index.isValid())
    {
        return QVariant();
    }
    if (index.row() >= fileList[0].size() || index.row() < 0)
    {
        return QVariant();
    }
    if (role == Qt::DisplayRole)
    {
        return fileList[index.column()][index.row()];
    }
    return QVariant();
}

bool FileListModel::canFetchMore(const QModelIndex &parent) const
{
    if (parent.isValid())
    {
        return false;
    }
    return (rowNumber < fileList[0].size());
}

void FileListModel::fetchMore(const QModelIndex &parent)
{
    if (parent.isValid())
    {
        return;
    }
    int remainder = fileList[0].size() - rowNumber;
    int itemsToFetch = qMin(100, remainder);

    if (itemsToFetch <= 0)
    {
        return;
    }
    beginInsertRows(QModelIndex(), rowNumber, rowNumber + itemsToFetch - 1);

    rowNumber += itemsToFetch;

    endInsertRows();
}

void FileListModel::setDataToTable(const QList<QList<float>> &data)
{
    beginResetModel();
    fileList = data;
    rowNumber = 0;
    endResetModel();
}

H FILE


#ifndef FILELISTMODEL_H
#define FILELISTMODEL_H

#include <QAbstractListModel>
#include <QStringList>

class FileListModel : public QAbstractTableModel
{
    Q_OBJECT

public:
    FileListModel(QObject *parent = nullptr);

    int rowCount(const QModelIndex &parent = QModelIndex()) const override;
    int columnCount(const QModelIndex &parent = QModelIndex()) const override;
    QVariant data(const QModelIndex &index, int role = Qt::DisplayRole) const override;
    void setColumnNumber(const int );
    void setDataToTable(const QList<QList<float>>&);

protected:
    bool canFetchMore(const QModelIndex &parent) const override;
    void fetchMore(const QModelIndex &parent) override;

private:
    QList<QList<float>> fileList;
    int rowNumber;
    int colNumber;
};

#endif // FILELISTMODEL_H

BUILDING MODEL

FileListModel *pModel =new FileListModel(this);
QList<QList<float>> a;
pModel->setColumnNumber(5);
a.append(qlist1);
a.append(qlist2);
a.append(qlist3);
a.append(qlist4);
a.append(qlist5);
pModel->setDataToTable(a);

Christian Ehrlicher

You're still using the wrong container. You want to use a vector, not a list. Your fetchMore() is trivial - I doubt this is your real workcase, or?

FroZtiZ

Why should I use a QVector instead of a QList?
My fetch is maybe trivial but it is what I was trying to do. I am not a QT expert, especially, I dont know at all the abstract classes.

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

Why should I use a QVector instead of a QList?

Because QList is a list (more or less) and QVector a vector. See also the QList documentation - since you're using floats, the memory needed for QList<float> is exactly double the memory needed for QVector<float> on 64 bit systems.

FroZtiZ

Thanks I will perform the modification

JonB

@FroZtiZ
Excuse my chiming in again. I have to confess I am lost as to whether you are now asking about which model to use, columns vs rows, efficiency of adding, or ...

@FroZtiZ about 18 hours ago

Minimum 5 columns of 20 000 rows

@Christian-Ehrlicher
20k items is not that larger that lazy loading is needed I would say

First, always heed what @Christian-Ehrlicher asks/suggests, as he knows more than I. However, I am slightly surprised, I would have said 20k (x 5 columns) is quite large. I don't know what your "very very slow" actually means. But my thought would be: if you are presenting a UI like a QTableWidget or whatever, why would you want 20k items in it? The user cannot handle that many items to scroll through....

FroZtiZ

I am doing what Christian suggested. And now, with the last modifications, everything is working. So I am not asking anymore about how to proceed with columns, rows or model.
When I say very very slow, I mean I need ~5 seconds to update/construct/destroy the table.
I agree that 100k items is quite large for a user to read everything. In fact, these items are spread in different tables that represent x,y coordinates of a function. For sure the user won't read everything simultaneously. However, I need to be able to provide the data on request inside the user interface.

JonB

@FroZtiZ
OK. Sorry if I am supposed to understand this from the foregoing, but I don't understand where your "~5 seconds to update/construct/destroy the table" is coming from? You have implemented a fetchMore() which only fetches 100 at a time, so that should be good. Is it being called repeatedly to read all 20/100k? Is there any overhead in physically getting the items? I don't understand where your 5 seconds is being spent? You need to discover that.

FroZtiZ

@FroZtiZ said in Large set of data in QTableView:

The construction and destruction of QStandardItem in a QTableWidget is very slow. That is the reason why I decided to construct a QAbstractTableModel.

My code was slow to execute when I was using a QTableWidget with QStandardItem.

@FroZtiZ said in Large set of data in QTableView:

The current model that is described in my code above is very fast so, it seems to be the solution. Inside, I want to put 5 QList, each being a column of this model.

@FroZtiZ said in Large set of data in QTableView:

Thank you for your contributions. I found how to proceed by storing my data in a QList<QList<float>>.
Please find below a working code that can probably be improved:

With the model I provided above, including "fetch", the problem is solved and the execution is fast.

Christian Ehrlicher

@FroZtiZ said in Large set of data in QTableView:

the problem is solved and the execution is fast.

Then please mark this topic as solved, thx.