QAbstractTableModel: Adding tousends of data items to table model results in frozen GUI

makopo

Hello everybody,

i have a very large set of data (up to 200.000 items) and I want to add the data to my table. Every item is a struct with 6 members:

struct DataItems 
{
    QString m_dataName;
    QString m_did;
    QString m_sdid;
    QString m_dbn;
    QVariant m_udw;
    int m_line;
};

Unfortunatly displaying data only works with a big latency and results in a freezing GUI. The curious thing is that I have a button that works before I add the data, but after adding data to the model the button is "frozen". I also have checkboxes and a combobox in my program and they works after data is added.
I have a function that is a slot and that receives the data and add it to the struct type.

void DataTableModel::addDataToTable(QString &name, QString &did, QString &sdid, QString &dbn, QVariant &udw, int &line)
{
    //if the parameter name is not in the table, add the whole data
    beginInsertRows(QModelIndex(), m_tableItems.count(), m_tableItems.count());
    DataItems items;
    items.m_dataName = name;
    items.m_did = did;
    items.m_sdid = sdid;
    items.m_dbn = dbn;
    items.m_udw = udw;
    items.m_line = line;
    m_tableItems.push_back(items);
    endInsertRows();
}

Is there a more efficient way to add data to the structre? If I add ~2000 items there is no problem with the latency. Furthermore the GUI and the class that sends the data run in different threads.

Any hint is appreciated.
Kind regards

JonB

@makopo
IMHO, adding "200.000 items" to an in-memory data client is "not a good idea". We have databases to hold this sort of data volume.... If you insist, there will be performance penalties. And if your UI shows all 200,000 items then even worse.

Your code shows adding a single row. The whole point of begin/endInsertRows() is to be able to optimize by allowing you to add a large number of rows in one go without the UI having to update. You are supposed to: beginInsertRows(), insert a whole bunch rows, endInsertRows(). If you have a large number to insert you should take advantage of this. And if you are completely filling the table with a large number then begin/endResetModel() may be even better.

Furthermore the GUI and the class that sends the data run in different threads.

This complicates things. If you are saying you cannot avoid that the "gather data thread" sends rows to be added one-at-a-time, and it's sending a large number of rows, and your "Ui thread" is presently adding them one-at-a-time as they arrive with the single-item beginInsertRows(), and you want to optimize this, then consider this.

Your UI does not need to "keep up" with updating for every single row when it arrives when you have such a large number. Batching these updates would allow you take advantage of multi-row inserts. Let the UI slot accumulate/buffer newly-received data rows. Have a timer which then adds from the buffer to the table however many have arrived after one second, or whatever. This could give you a big performance increase, depending on your situation.

makopo

@JonB
Thanks for your advice. Of course the app should run with a good perfomance. So pushing the data into DB and read from it sounds like a good idea.

Also adding a "bigger bunch" of data to the model like this, shows a better performance. But it is also not good.

void DataTableModel::addDataToTable(QString &name, QString &did, QString sdid, QString dbn, QVariant &udw, int &line)
{
    beginInsertRows(QModelIndex(), m_tableItems.count(), m_tableItems.count());
    DataItems items;
    items.m_dataName = name;
    items.m_did = did;
    items.m_sdid = sdid;
    items.m_dbn = dbn;
    items.m_udw = udw;
    items.m_line = line;
    endInsertRows();
    m_tableItems.push_back(items);
}

Thank you for showing me the aspects.

JonB

@makopo said in QAbstractTableModel: Adding tousends of data items to table model results in frozen GUI:

Also adding a "bigger bunch" of data to the model like this, shows a better performance. But it is also not good.

Your code is identical to before, not changed to insert multiple rows at a time.