Fast search through QTableWidget rows

mardzo · wrote on 14 Dec 2016, 19:01

Hello
I need to search rows through a qtablewidget. Each of the rows in the table contains a field with date and I need to show only rows that are within a specified date interval based on user input. Here is my function:

void nvr::sort()
{

    QTableWidget* tabela = this->findChild<QTableWidget*>("NCtable");



    QDateEdit* c1 = this->findChild<QDateEdit*>("c1");

    QDateEdit* c2 = this->findChild<QDateEdit*>("c2");

    // user specified ranges for date
    QDate date1 = c1->date();

    QDate date2 = c2->date();

    //row numbers in table
    int rowsNum = tabela->rowCount();

    // hide all rows
    for(int z = 0; z < rowsNum; z++) {

        tabela->hideRow(z);
   
    }

    // show only rows that are within range
    for(int z = 0; z < rowsNum; z++) {

        
        QDateTime dateTime = QDateTime::fromString(tabela->item(z,2)->text(),"dd.MM.yyyy hh:mm");

        QDate date = dateTime.date();

        //date compares
        if ( (date1.operator <=(date)) && (date2.operator >=(date) ) ) {

        tabela->showRow(z);

        }

 

       }



}

This works fine if i have 200 rows. But when i have 30 000 rows and i surely will, the gui freezes because i suppose the function executes very slow. Any suggestions for faster execution?

Chris Kawa · 15 Dec 2016, 07:40

A couple of suggestions:

Don't do two passes to hide all and then show only some. Just loop once and use setRowHidden(row, true/false).
Using string operations to compare dates is very inefficient. Instead of that store a date as a date in some user role of the item and get it back directly i.e.:

//when you add an item:
QDate d = ... //whatever 
QTableWidgetItem* item = new QTableWidgetItem(d.toString("dd.MM.yyyy"));
item->setData(Qt::UserRole, d); //store the date in custom role as date not as text

and then when you compare them:

QDate date = tabela->item(z,2)->data(Qt::UserRole).toDate();

This way you avoid a lot of slooow conversions.

What's with the explicit date1.operator <=(date)? Why not just date1 < date?

Other than that you should consider switching from QTableWidget to QTableView + QSortFilterProxyModel + QStandardItemModel. Proxy models are better for filtering tasks as they bulk their operations.

mardzo · wrote on 15 Dec 2016, 07:40

@Chris-Kawa said in Fast search through QTableWidget rows:

QSortFilterProxyModel

Thank for your answer. I have switched to QTableView with a standard model and i cant figure out how to use QSortFilterProxy model. Any suggestions?

VRonin · wrote on 15 Dec 2016, 08:06

either you use the QAbstarctItemModel interface model->setData(model->index(row,column),d,Qt::UserRole) or you just act on the underlying QStandardItemModel and use item->setData(Qt::UserRole, d); on it

mardzo · wrote on 16 Dec 2016, 18:22

Okay, so I have implemented the QSortFilterProxyModel and the filtering is much faster like this. I have another question and that is how to convert this model to html table so i can export it as pdf? Looping with a for loop between the data in the rows in order to create html is fine when i have 1000 rows, but when i have 6000 rows i have to wait 2-3 minutes in order to create the html. Any suggestions?

Chris Kawa · wrote on 16 Dec 2016, 18:43

Minutes? There's got to be something wrong with how you generate the html.
Consider this code:

    QElapsedTimer timer;
    timer.start();

    QStandardItemModel model;
    for (int i = 0; i < 100000; ++i) {
         QDate d = QDate::fromJulianDay(qrand());
         QStandardItem* item = new QStandardItem(d.toString("dd.MM.yyyy"));
         item->setData(d, Qt::UserRole);
         model.appendRow(item);
    }

    qDebug() << timer.restart() << "ms";

    QString html = "<html><body><table>";
    html.reserve(40 * model.rowCount()); //40 is some heuristic value
    for (int i = 0; i < model.rowCount(); ++i)
    {
        html += "<tr><td>";
        html += model.data(model.index(i, 0), Qt::DisplayRole).toString();
        html += "</td></tr>";
    }
    html += "</table></body></html>";

    qDebug() << timer.restart() << "ms";

For 100 000 elements it takes on my machine 230 milliseconds to create the model and 40ms to generate the html.

Btw. Make sure you're not doing measurements in Debug mode.

mardzo · wrote on 16 Dec 2016, 19:21

Ok so this is my code and im trying in release mode. Im sorry for the bad formatting Still its very slow.


void AccControl::exportToPdf()  {


    ACCSortFilterProxyModel* proxyModel = this->findChild<ACCSortFilterProxyModel*>("proxy");

    QComboBox* searchCombo = this->findChild<QComboBox*>("searchCombo");

    QDateEdit* c1 = this->findChild<QDateEdit*>("c1");

    QDateEdit* c2 = this->findChild<QDateEdit*>("c2");

    QString tablecss = " style = 'margin-top:10px; margin-right:20px; border-collapse: separate; background-color: #000;'";

    QString tdthcss = " style = 'text-align: left; padding: 1px; background-color: #fff; font-size:6pt; ' ";

    QString htmltable = "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0//EN\" \"http://www.w3.org/TR/REC-html40/strict.dtd\">"
                                                 "<html><head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"><meta name=\"qrichtext\" content=\"1\">"
                                                 "<title>QTextEdit Example</title>"
                                                 "<style type=\"text/css\">p, li { white-space: pre-wrap; }"
                                                 "</style></head><body style=\" font-family:Tahoma; font-size:9pt; font-weight:300; font-"
                                                 "style:normal;\">" "<table align=center style = 'margin-top:50px;  margin-right:20px; '><tr><th><h2><span>Извештај за настани од контрола на пристап</span></h2></th></tr></table><table align=center style='margin-top:10px;  margin-right:20px; '><tr><th"+ tdthcss +"><span>Филтер:</span></th><th"+ tdthcss +"><span>" +  searchCombo->currentText() + "</span></th><th"+ tdthcss +"><span>Од:</span></th><th"+ tdthcss +"><span>"+ c1->date().toString("dd.MM.yyyy") +"</span></th><th"+ tdthcss +"><span>До:</span></th><th"+ tdthcss +"><span>"+ c2->date().toString("dd.MM.yyyy") +"</span></th></tr></table>";

    htmltable= htmltable + "<table align=center cellspacing=1 width=440 " + tablecss + "> <tr> <th" + tdthcss  + "><span>Бр.</span></th> <th" + tdthcss  + "><span>Уред</span></th> <th" + tdthcss  + "><span>Лог</span></th> <th" + tdthcss  + "><span>Врата</span></th> <th" + tdthcss  + "><span>UID</span></th> <th" + tdthcss  + "><span>Корисник</span></th> <th" + tdthcss  + "><span>Датум и Време</span></th> </tr>";


    int j = 1;

    int k = 1;

    QStringList split1;
    QStringList split2;

    for (int i =0; i < proxyModel->rowCount(); i++) {



            //ova za koloritnost na redici
           if ((j%2)!=0) { tdthcss = " style = 'text-align: left; padding: 1px; background-color: #dddddd; font-size:6pt;' "; } else { tdthcss = " style = 'text-align: left; padding: 1px; background-color: #fff; font-size:6pt;' "; }

           //za reden broj

           htmltable = htmltable+"<tr> <td" + tdthcss  + "><span>" + QString::number(j) + "</span></td>";


            //za device

            htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,0).data(Qt::DisplayRole).toString() + "</span></td>";


            //za log

            htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,1).data(Qt::DisplayRole).toString() + "</span></td>";

            //za vrata

            htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,2).data(Qt::DisplayRole).toString() + "</span></td>";

            //za uid go zakomentirav jer realno ne se sobira na A4  dokolku go vrakam treba da ja dodadam kolonatai i vo html tabelata

            //pravam splitovi da go izvadam samo edniot tip na prikaz kartica

             if(proxyModel->index(i,3).data(Qt::DisplayRole).toString().contains('(')) {

             split1 = proxyModel->index(i,3).data(Qt::DisplayRole).toString().split('(');

             // ovaj if e za zastita da ne vlezam u nedefiniran element na lista
             if(split1.length()==2) {

             split2 = split1[1].split(')');

             htmltable = htmltable+"<td" + tdthcss  + ">" + split2[0] + "</td>";

             } else {htmltable = htmltable+"<td" + tdthcss  + ">" + "</td>";}

             split1.clear();
             split2.clear();

             }
             else { htmltable = htmltable+"<td" + tdthcss  + ">" + "</td>";}



            //za name

            htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,4).data(Qt::DisplayRole).toString() + "</span></td>";


            //za datum

            htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,5).data(Qt::DisplayRole).toString() + "</span></td></tr>";

            j++;
            k++;

            //za da ja seckam na stranici
            if (k == 51) {

            k = 1;

            htmltable = htmltable + "</table> </body> </html>";

            htmltable = htmltable + "<html  > <head>  </head> <body style = ''><table align=center style = 'margin-top:50px;  margin-right:20px; '><tr><th><h2><span>Извештај за настани од контрола на пристап</h2></span></th></tr></table><table align=center style='margin-top:10px;  margin-right:20px; '><tr><th"+ tdthcss +"><span>Филтер:</span></th><th"+ tdthcss +"><span>" +  searchCombo->currentText() + "</span></th><th"+ tdthcss +"><span>Од:</span></th><th"+ tdthcss +"><span>"+ c1->date().toString("dd.MM.yyyy") +"</span></th><th"+ tdthcss +"><span>До:</span></th><th"+ tdthcss +"><span>"+ c2->date().toString("dd.MM.yyyy") +"</span></th></tr></table>";

            htmltable= htmltable + "<table align=center cellspacing=1 width=440 " + tablecss + "> <tr> <th" + tdthcss  + "><span>Бр.</span></th> <th" + tdthcss  + "><span>Уред</span></th> <th" + tdthcss  + "><span>Лог</span></th> <th" + tdthcss  + "><span>Врата</span></th> <th" + tdthcss  + "><span>UID</span></th><th" + tdthcss  + "><span>Корисник</span></th> <th" + tdthcss  + "><span>Датум и Време</span></th> </tr>";


            };


    }

    htmltable = htmltable + "</table> </body> </html>";

    QString fileName = QFileDialog::getSaveFileName(this,tr("Снимај листа на логови како PDF"),"", tr("PDF files (*.pdf)"));

        if (fileName.isEmpty())

            return;




        QPrinter printer;

        printer.setResolution(QPrinter::HighResolution);

        printer.setOutputFormat(QPrinter::PdfFormat);

        printer.setOutputFileName(fileName);

        // printer.setFullPage(true);

        printer.setOrientation(QPrinter::Portrait);

        printer.setPaperSize(QPrinter::A4);

        QSizeF size = printer.paperSize(QPrinter::Point);

       // QTextEdit *edit = new QTextEdit(this);

        //edit->setHtml(htmltable);

        QTextDocument doc;

        doc.setHtml(htmltable);

        doc.setPageSize(size);

        doc.print(&printer);


}

Chris Kawa · wrote on 16 Dec 2016, 19:47

The problem in this code is the classical one - it's that there's no one super-slow place in it. It's that all of it is kinda slow in small amounts and it just piles up.
Couple of examples that hopefully you can expand on:

if ((j%2)!=0) { tdthcss = " style = 'text-align: left; padding: 1px; background-color: #dddddd; font-size:6pt;' "; }
        else { tdthcss = " style = 'text-align: left; padding: 1px; background-color: #fff; font-size:6pt;' "; }

Thousands of times you recreate new string tdthcss, while you could just once create both versions as separate constants and use one or the other.

htmltable = htmltable+"<tr> <td" + tdthcss  + "><span>" + QString::number(j) + "</span></td>";

Each + and = here is a costly reallocation:

htmltable+"<tr> <td" //allocates a temporary and copies both strings into it
+ tdthcss // reallocates the temporary to make room and copies
+ "><span>" //reallocates the temporary to make room and copies
+ QString::number(j) // I hope you see where this is going...

Instead of doing all those reallocations create the string once, reserve some expected amount of storage and then add to it:

htmlTable.reserve(<whatever you think is gonna be enouh>);
for (...) {
   htmlTable += "<tr> <td"; //no reallocations
   htmlTable += tdthcss; //no reallocations
   ...
}

On to the next lines:

if(proxyModel->index(i,3).data(Qt::DisplayRole).toString().contains('(')) {
            split1 = proxyModel->index(i,3).data(Qt::DisplayRole).toString().split('(');

Conversions do cost. Don't do them twice if you don't have to:

QString foo = proxyModel->index(i,3).data(Qt::DisplayRole).toString();
if(foo.contains('(')) {
            split1 = foo.split('(');

Next - the splits. Splitting string is very costly. It creates copies of each part and QStringList is not a great structure either. Instead use splitRef, which creates a nice vector of small QStringRef's. These are just pointers into the existing data so no character copying is made at all.

Better yet - why split at all? What you really want to do is find text between ( and ) so why allocate any dynamic container (even for refs).
Here's how you could just find the fragment in-place without any dynamic allocations:

QString foo = proxyModel->index(i,3).data(Qt::DisplayRole).toString();
int start = foo.indexOf('('); //just search no allocations
if (start >= 0) {
   int end = foo.indexOf(')', start); //just search no allocations
   if (end >=0)
      htmlTable += foo.midRef(start, end - start); //just copy, no reallocations
   else ...
}
else ...

And finally

split1.clear();
split2.clear();

That's just waste of time. You're reassigning the variables each time so clearing them afterwards is just wasted cycles.

hskoglund · wrote on 16 Dec 2016, 20:19

Hi, just want to add to @Chris-Kawa's great advice, to ease the burden on that poor QString htmltable that gets reassigned lots of times, you could try using a local QString inside your loop, like this:

for (int i =0; i < proxyModel->rowCount(); i++) {
    ...
    //za reden broj
    QString htmlrow;
    htmlrow.reserve(1337);
    htmlrow = "<tr> <td" + tdthcss  + "><span>" + QString::number(j) + "</span></td>";
    
    //za device
    htmlrow += "<td" + tdthcss  + "><span>" + proxyModel->index(i,0).data(Qt::DisplayRole).toString() + "</span></td>";
    ...

then at the end of the loop you add htmlrow to htmltable

    htmlrow += "<table align=center cellspacing=1 width=440 " + tablecss + "> <tr> <th" + tdthcss  + "><span>Бр.</span></th> <th" + tdthcss  + "><span>Уред</span></th> <th" + tdthcss  + "><span>Лог</span></th> <th" + tdthcss  + "><span>Врата</span></th> <th" + tdthcss  + "><span>UID</span></th><th" + tdthcss  + "><span>Корисник</span></th> <th" + tdthcss  + "><span>Датум и Време</span></th> </tr>";
    };
    
    htmltable += htmlrow;
 }

mardzo · wrote on 17 Dec 2016, 11:04

Thanks guys for your advice. Your optimisations increased the generation speed of my table a lot. The only thing that seems still to be slow is the QPrinter when table has thousands of rows. Should i move the printer in a different thread in order to prevent gui freezing?

  QPrinter printer;

        printer.setResolution(QPrinter::HighResolution);

        printer.setOutputFormat(QPrinter::PdfFormat);

        printer.setOutputFileName(fileName);

        // printer.setFullPage(true);

        printer.setOrientation(QPrinter::Portrait);

        printer.setPaperSize(QPrinter::A4);

        QSizeF size = printer.paperSize(QPrinter::Point);

       // QTextEdit *edit = new QTextEdit(this);

        //edit->setHtml(htmltable);

        QTextDocument doc;

        doc.setHtml(htmltable);

        doc.setPageSize(size);

        doc.print(&printer);

mardzo · wrote on 17 Dec 2016, 11:17

debug shows that it is not actually the printer that is making the gui freezes. It is the line that sets the html to the QTextDocument that is slowing down. Any suggestions?

doc.setHtml(htmltable);

Chris Kawa · wrote on 17 Dec 2016, 11:53

I guess you're doing the work twice. Instead of building html string and then parse it to convert to QTextDocument maybe you could just build the QTextDocument directly using QTextCursor. That should be faster.

mardzo · wrote on 17 Dec 2016, 13:02

Thanks for the suggestion but as i already have a worker thread used for stg else, i am now printing in the worker thread and im sending the string via a signal.