[solved] tableWidget building time 45sec (85000 rows, 1 for loop). How to optimize build time?

[victoryfox]

Greatings everyone,

I have a little qt project, a LogViewer : reads a .log file (wich can have cca 100-150mb: about million+ lines), parses the file and then builds the gui.

So you see, the app is divided in two parts (parsing data & filling the table). I have done that to see which part needs to be optimized better. I have no backend database, only one colleciton of objects.

My two main methods are:

void ParseLog(QString strInputFile); -> parser
void RefreshLog(); -> tableWidget builder

Parser has only one loop (while) - reads the entire file line by line and parses needed data
Builder has only one loop (for) - for every object i have in my collection stored, create a row

My parsing method for a file of 130mb (cca 1.222.212 lines) lasts about 20sec, which is ok, eventually i will optimize the parser further.

The problem is the GUI building method "RefreshLog()"

@
void Logviewer::RefreshLog()
{
startGui = QDateTime::currentDateTime();
int nlastRow = 0;
QTableWidgetItem *itemId;
QTableWidgetItem *itemCode;
QTableWidgetItem *itemCallResult;
QTableWidgetItem *itemClientIP;
QTableWidgetItem *itemServerIP;
QTableWidgetItem *itemCallLength;
QTableWidgetItem *itemDate;

for(int i=0; i<m_lstLogData.size(); i++)
{

itemId = new QTableWidgetItem();
itemCode = new QTableWidgetItem();
itemCallResult = new QTableWidgetItem();
itemClientIP = new QTableWidgetItem();
itemServerIP = new QTableWidgetItem();
itemCallLength = new QTableWidgetItem();
itemDate = new QTableWidgetItem();

itemId->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strID));
itemCode->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strCode));
itemCallResult->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strResult));
itemClientIP->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strClientIP));
itemServerIP->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strServerIP));
itemCallLength->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strCallLength));
itemDate->setData(Qt::DisplayRole, QVariant(m_lstLogData[i].strDate));

ui.tableWidget->insertRow(nlastRow);

ui.tableWidget->setItem(nlastRow,0, itemId);
ui.tableWidget->setItem(nlastRow,1, itemCode);
ui.tableWidget->setItem(nlastRow,2, itemCallResult);
ui.tableWidget->setItem(nlastRow,3, itemClientIP);
ui.tableWidget->setItem(nlastRow,4, itemServerIP);
ui.tableWidget->setItem(nlastRow,5, itemCallLength);
ui.tableWidget->setItem(nlastRow,6, itemDate);

nlastRow++;
}
endGui = QDateTime::currentDateTime();
}@

This function right above takes about 45 seconds for 85000 rows, 85000 loops and for every loop doing:

• new QTableWidgetItem()
• setData
• insertRow
• setItem

Which i believe some cost too much.

Can you please give me advice on how to optimize this, or should i move to QTableView?

I haven't done model-view programming, so if it is gonna be much faster and better, i will need to switch then.

Any advice for mobel-view for my project will be helpfull, i will start with some model-view tutorials for now ("should this be correct materials for my need?":http://qt.nokia.com/learning/topics/modelview)

Or can i stay with the simplier widget and optimize it a little?

THANKS
victoryfox

[loladiro]

For data sets that large Model/View is the only way. There is a nice tutorial on the "docs":http://doc.qt.nokia.com/4.7/modelview.html

[giesbert]

You should definitely switch to model/view.
We also have some log tools here and we build them completely by model/view (but it's closed source...). Just take care that you insert all rows and then update the view and do not update the view after each row. otherwise you will have the same problems as now. Adding data to your internal storage in the model will be fast, UI updates are cost intensive.

Your internal model might directly use your existing structure (m_lstLogData) and just give access to it, that's what we do with huge models.

[victoryfox]

[quote author="loladiro" date="1312450273"]For data sets that large Model/View is the only way. There is a nice tutorial on the "docs":http://doc.qt.nokia.com/4.7/modelview.html[/quote]

thanks

[victoryfox]

[quote author="Gerolf" date="1312452929"]You should definitely switch to model/view.
We also have some log tools here and we build them completely by model/view (but it's closed source...). Just take care that you insert all rows and then update the view and do not update the view after each row. otherwise you will have the same problems as now. Adding data to your internal storage in the model will be fast, UI updates are cost intensive.

Your internal model might directly use your existing structure (m_lstLogData) and just give access to it, that's what we do with huge models.[/quote]

Then i will go with model-view implementation. I'm not sure if the automatic sorting capabilities of the column headers are any good for a large number of rows, but in the tableWidget you wish you have not pressed the header cell for sorting :) - maybe i will implement a custom "Filter & Search" option lateron in model-view.

Thanks for the good advices. Do i need to tag this topic as closed or something?

Greetings,
victoryfox

[Eddy]

bq. Do i need to tag this topic as closed or something?

Yes, please could you edit your title and add [solved] in front of it. There are no other means at this moment, but the Trolls are implementing a way now.