Fast search through QTableWidget rows



  • Hello
    I need to search rows through a qtablewidget. Each of the rows in the table contains a field with date and I need to show only rows that are within a specified date interval based on user input. Here is my function:

    void nvr::sort()
    {
    
        QTableWidget* tabela = this->findChild<QTableWidget*>("NCtable");
    
    
    
        QDateEdit* c1 = this->findChild<QDateEdit*>("c1");
    
        QDateEdit* c2 = this->findChild<QDateEdit*>("c2");
    
        // user specified ranges for date
        QDate date1 = c1->date();
    
        QDate date2 = c2->date();
    
        //row numbers in table
        int rowsNum = tabela->rowCount();
    
        // hide all rows
        for(int z = 0; z < rowsNum; z++) {
    
            tabela->hideRow(z);
       
        }
    
        // show only rows that are within range
        for(int z = 0; z < rowsNum; z++) {
    
            
            QDateTime dateTime = QDateTime::fromString(tabela->item(z,2)->text(),"dd.MM.yyyy hh:mm");
    
            QDate date = dateTime.date();
    
            //date compares
            if ( (date1.operator <=(date)) && (date2.operator >=(date) ) ) {
    
            tabela->showRow(z);
    
            }
    
     
    
           }
    
    
    
    }
    

    This works fine if i have 200 rows. But when i have 30 000 rows and i surely will, the gui freezes because i suppose the function executes very slow. Any suggestions for faster execution?


  • Moderators

    A couple of suggestions:

    • Don't do two passes to hide all and then show only some. Just loop once and use setRowHidden(row, true/false).
    • Using string operations to compare dates is very inefficient. Instead of that store a date as a date in some user role of the item and get it back directly i.e.:
    //when you add an item:
    QDate d = ... //whatever 
    QTableWidgetItem* item = new QTableWidgetItem(d.toString("dd.MM.yyyy"));
    item->setData(Qt::UserRole, d); //store the date in custom role as date not as text
    

    and then when you compare them:

    QDate date = tabela->item(z,2)->data(Qt::UserRole).toDate();
    

    This way you avoid a lot of slooow conversions.

    • What's with the explicit date1.operator <=(date)? Why not just date1 < date?

    Other than that you should consider switching from QTableWidget to QTableView + QSortFilterProxyModel + QStandardItemModel. Proxy models are better for filtering tasks as they bulk their operations.



  • @Chris-Kawa said in Fast search through QTableWidget rows:

    QSortFilterProxyModel

    Thank for your answer. I have switched to QTableView with a standard model and i cant figure out how to use QSortFilterProxy model. Any suggestions?



  • either you use the QAbstarctItemModel interface model->setData(model->index(row,column),d,Qt::UserRole) or you just act on the underlying QStandardItemModel and use item->setData(Qt::UserRole, d); on it



  • Okay, so I have implemented the QSortFilterProxyModel and the filtering is much faster like this. I have another question and that is how to convert this model to html table so i can export it as pdf? Looping with a for loop between the data in the rows in order to create html is fine when i have 1000 rows, but when i have 6000 rows i have to wait 2-3 minutes in order to create the html. Any suggestions?


  • Moderators

    Minutes? There's got to be something wrong with how you generate the html.
    Consider this code:

        QElapsedTimer timer;
        timer.start();
    
        QStandardItemModel model;
        for (int i = 0; i < 100000; ++i) {
             QDate d = QDate::fromJulianDay(qrand());
             QStandardItem* item = new QStandardItem(d.toString("dd.MM.yyyy"));
             item->setData(d, Qt::UserRole);
             model.appendRow(item);
        }
    
        qDebug() << timer.restart() << "ms";
    
        QString html = "<html><body><table>";
        html.reserve(40 * model.rowCount()); //40 is some heuristic value
        for (int i = 0; i < model.rowCount(); ++i)
        {
            html += "<tr><td>";
            html += model.data(model.index(i, 0), Qt::DisplayRole).toString();
            html += "</td></tr>";
        }
        html += "</table></body></html>";
    
        qDebug() << timer.restart() << "ms";
    

    For 100 000 elements it takes on my machine 230 milliseconds to create the model and 40ms to generate the html.

    Btw. Make sure you're not doing measurements in Debug mode.



  • Ok so this is my code and im trying in release mode. Im sorry for the bad formatting Still its very slow.

    
    void AccControl::exportToPdf()  {
    
    
        ACCSortFilterProxyModel* proxyModel = this->findChild<ACCSortFilterProxyModel*>("proxy");
    
        QComboBox* searchCombo = this->findChild<QComboBox*>("searchCombo");
    
        QDateEdit* c1 = this->findChild<QDateEdit*>("c1");
    
        QDateEdit* c2 = this->findChild<QDateEdit*>("c2");
    
        QString tablecss = " style = 'margin-top:10px; margin-right:20px; border-collapse: separate; background-color: #000;'";
    
        QString tdthcss = " style = 'text-align: left; padding: 1px; background-color: #fff; font-size:6pt; ' ";
    
        QString htmltable = "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.0//EN\" \"http://www.w3.org/TR/REC-html40/strict.dtd\">"
                                                     "<html><head><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"><meta name=\"qrichtext\" content=\"1\">"
                                                     "<title>QTextEdit Example</title>"
                                                     "<style type=\"text/css\">p, li { white-space: pre-wrap; }"
                                                     "</style></head><body style=\" font-family:Tahoma; font-size:9pt; font-weight:300; font-"
                                                     "style:normal;\">" "<table align=center style = 'margin-top:50px;  margin-right:20px; '><tr><th><h2><span>Извештај за настани од контрола на пристап</span></h2></th></tr></table><table align=center style='margin-top:10px;  margin-right:20px; '><tr><th"+ tdthcss +"><span>Филтер:</span></th><th"+ tdthcss +"><span>" +  searchCombo->currentText() + "</span></th><th"+ tdthcss +"><span>Од:</span></th><th"+ tdthcss +"><span>"+ c1->date().toString("dd.MM.yyyy") +"</span></th><th"+ tdthcss +"><span>До:</span></th><th"+ tdthcss +"><span>"+ c2->date().toString("dd.MM.yyyy") +"</span></th></tr></table>";
    
        htmltable= htmltable + "<table align=center cellspacing=1 width=440 " + tablecss + "> <tr> <th" + tdthcss  + "><span>Бр.</span></th> <th" + tdthcss  + "><span>Уред</span></th> <th" + tdthcss  + "><span>Лог</span></th> <th" + tdthcss  + "><span>Врата</span></th> <th" + tdthcss  + "><span>UID</span></th> <th" + tdthcss  + "><span>Корисник</span></th> <th" + tdthcss  + "><span>Датум и Време</span></th> </tr>";
    
    
        int j = 1;
    
        int k = 1;
    
        QStringList split1;
        QStringList split2;
    
        for (int i =0; i < proxyModel->rowCount(); i++) {
    
    
    
                //ova za koloritnost na redici
               if ((j%2)!=0) { tdthcss = " style = 'text-align: left; padding: 1px; background-color: #dddddd; font-size:6pt;' "; } else { tdthcss = " style = 'text-align: left; padding: 1px; background-color: #fff; font-size:6pt;' "; }
    
               //za reden broj
    
               htmltable = htmltable+"<tr> <td" + tdthcss  + "><span>" + QString::number(j) + "</span></td>";
    
    
                //za device
    
                htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,0).data(Qt::DisplayRole).toString() + "</span></td>";
    
    
                //za log
    
                htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,1).data(Qt::DisplayRole).toString() + "</span></td>";
    
                //za vrata
    
                htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,2).data(Qt::DisplayRole).toString() + "</span></td>";
    
                //za uid go zakomentirav jer realno ne se sobira na A4  dokolku go vrakam treba da ja dodadam kolonatai i vo html tabelata
    
                //pravam splitovi da go izvadam samo edniot tip na prikaz kartica
    
                 if(proxyModel->index(i,3).data(Qt::DisplayRole).toString().contains('(')) {
    
                 split1 = proxyModel->index(i,3).data(Qt::DisplayRole).toString().split('(');
    
                 // ovaj if e za zastita da ne vlezam u nedefiniran element na lista
                 if(split1.length()==2) {
    
                 split2 = split1[1].split(')');
    
                 htmltable = htmltable+"<td" + tdthcss  + ">" + split2[0] + "</td>";
    
                 } else {htmltable = htmltable+"<td" + tdthcss  + ">" + "</td>";}
    
                 split1.clear();
                 split2.clear();
    
                 }
                 else { htmltable = htmltable+"<td" + tdthcss  + ">" + "</td>";}
    
    
    
                //za name
    
                htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,4).data(Qt::DisplayRole).toString() + "</span></td>";
    
    
                //za datum
    
                htmltable = htmltable+"<td" + tdthcss  + "><span>" + proxyModel->index(i,5).data(Qt::DisplayRole).toString() + "</span></td></tr>";
    
                j++;
                k++;
    
                //za da ja seckam na stranici
                if (k == 51) {
    
                k = 1;
    
                htmltable = htmltable + "</table> </body> </html>";
    
                htmltable = htmltable + "<html  > <head>  </head> <body style = ''><table align=center style = 'margin-top:50px;  margin-right:20px; '><tr><th><h2><span>Извештај за настани од контрола на пристап</h2></span></th></tr></table><table align=center style='margin-top:10px;  margin-right:20px; '><tr><th"+ tdthcss +"><span>Филтер:</span></th><th"+ tdthcss +"><span>" +  searchCombo->currentText() + "</span></th><th"+ tdthcss +"><span>Од:</span></th><th"+ tdthcss +"><span>"+ c1->date().toString("dd.MM.yyyy") +"</span></th><th"+ tdthcss +"><span>До:</span></th><th"+ tdthcss +"><span>"+ c2->date().toString("dd.MM.yyyy") +"</span></th></tr></table>";
    
                htmltable= htmltable + "<table align=center cellspacing=1 width=440 " + tablecss + "> <tr> <th" + tdthcss  + "><span>Бр.</span></th> <th" + tdthcss  + "><span>Уред</span></th> <th" + tdthcss  + "><span>Лог</span></th> <th" + tdthcss  + "><span>Врата</span></th> <th" + tdthcss  + "><span>UID</span></th><th" + tdthcss  + "><span>Корисник</span></th> <th" + tdthcss  + "><span>Датум и Време</span></th> </tr>";
    
    
                };
    
    
        }
    
        htmltable = htmltable + "</table> </body> </html>";
    
        QString fileName = QFileDialog::getSaveFileName(this,tr("Снимај листа на логови како PDF"),"", tr("PDF files (*.pdf)"));
    
            if (fileName.isEmpty())
    
                return;
    
    
    
    
            QPrinter printer;
    
            printer.setResolution(QPrinter::HighResolution);
    
            printer.setOutputFormat(QPrinter::PdfFormat);
    
            printer.setOutputFileName(fileName);
    
            // printer.setFullPage(true);
    
            printer.setOrientation(QPrinter::Portrait);
    
            printer.setPaperSize(QPrinter::A4);
    
            QSizeF size = printer.paperSize(QPrinter::Point);
    
           // QTextEdit *edit = new QTextEdit(this);
    
            //edit->setHtml(htmltable);
    
            QTextDocument doc;
    
            doc.setHtml(htmltable);
    
            doc.setPageSize(size);
    
            doc.print(&printer);
    
    
    }
    

  • Moderators

    The problem in this code is the classical one - it's that there's no one super-slow place in it. It's that all of it is kinda slow in small amounts and it just piles up.
    Couple of examples that hopefully you can expand on:

    if ((j%2)!=0) { tdthcss = " style = 'text-align: left; padding: 1px; background-color: #dddddd; font-size:6pt;' "; }
            else { tdthcss = " style = 'text-align: left; padding: 1px; background-color: #fff; font-size:6pt;' "; }
    

    Thousands of times you recreate new string tdthcss, while you could just once create both versions as separate constants and use one or the other.

    htmltable = htmltable+"<tr> <td" + tdthcss  + "><span>" + QString::number(j) + "</span></td>";
    

    Each + and = here is a costly reallocation:

    htmltable+"<tr> <td" //allocates a temporary and copies both strings into it
    + tdthcss // reallocates the temporary to make room and copies
    + "><span>" //reallocates the temporary to make room and copies
    + QString::number(j) // I hope you see where this is going...
    

    Instead of doing all those reallocations create the string once, reserve some expected amount of storage and then add to it:

    htmlTable.reserve(<whatever you think is gonna be enouh>);
    for (...) {
       htmlTable += "<tr> <td"; //no reallocations
       htmlTable += tdthcss; //no reallocations
       ...
    }
    

    On to the next lines:

    if(proxyModel->index(i,3).data(Qt::DisplayRole).toString().contains('(')) {
                split1 = proxyModel->index(i,3).data(Qt::DisplayRole).toString().split('(');
    

    Conversions do cost. Don't do them twice if you don't have to:

    QString foo = proxyModel->index(i,3).data(Qt::DisplayRole).toString();
    if(foo.contains('(')) {
                split1 = foo.split('(');
    

    Next - the splits. Splitting string is very costly. It creates copies of each part and QStringList is not a great structure either. Instead use splitRef, which creates a nice vector of small QStringRef's. These are just pointers into the existing data so no character copying is made at all.

    Better yet - why split at all? What you really want to do is find text between ( and ) so why allocate any dynamic container (even for refs).
    Here's how you could just find the fragment in-place without any dynamic allocations:

    QString foo = proxyModel->index(i,3).data(Qt::DisplayRole).toString();
    int start = foo.indexOf('('); //just search no allocations
    if (start >= 0) {
       int end = foo.indexOf(')', start); //just search no allocations
       if (end >=0)
          htmlTable += foo.midRef(start, end - start); //just copy, no reallocations
       else ...
    }
    else ...
    

    And finally

    split1.clear();
    split2.clear();
    

    That's just waste of time. You're reassigning the variables each time so clearing them afterwards is just wasted cycles.



  • Hi, just want to add to @Chris-Kawa's great advice, to ease the burden on that poor QString htmltable that gets reassigned lots of times, you could try using a local QString inside your loop, like this:

    for (int i =0; i < proxyModel->rowCount(); i++) {
        ...
        //za reden broj
        QString htmlrow;
        htmlrow.reserve(1337);
        htmlrow = "<tr> <td" + tdthcss  + "><span>" + QString::number(j) + "</span></td>";
        
        //za device
        htmlrow += "<td" + tdthcss  + "><span>" + proxyModel->index(i,0).data(Qt::DisplayRole).toString() + "</span></td>";
        ...
    

    then at the end of the loop you add htmlrow to htmltable

        htmlrow += "<table align=center cellspacing=1 width=440 " + tablecss + "> <tr> <th" + tdthcss  + "><span>Бр.</span></th> <th" + tdthcss  + "><span>Уред</span></th> <th" + tdthcss  + "><span>Лог</span></th> <th" + tdthcss  + "><span>Врата</span></th> <th" + tdthcss  + "><span>UID</span></th><th" + tdthcss  + "><span>Корисник</span></th> <th" + tdthcss  + "><span>Датум и Време</span></th> </tr>";
        };
        
        htmltable += htmlrow;
     }


  • Thanks guys for your advice. Your optimisations increased the generation speed of my table a lot. The only thing that seems still to be slow is the QPrinter when table has thousands of rows. Should i move the printer in a different thread in order to prevent gui freezing?

      QPrinter printer;
    
            printer.setResolution(QPrinter::HighResolution);
    
            printer.setOutputFormat(QPrinter::PdfFormat);
    
            printer.setOutputFileName(fileName);
    
            // printer.setFullPage(true);
    
            printer.setOrientation(QPrinter::Portrait);
    
            printer.setPaperSize(QPrinter::A4);
    
            QSizeF size = printer.paperSize(QPrinter::Point);
    
           // QTextEdit *edit = new QTextEdit(this);
    
            //edit->setHtml(htmltable);
    
            QTextDocument doc;
    
            doc.setHtml(htmltable);
    
            doc.setPageSize(size);
    
            doc.print(&printer);
    


  • debug shows that it is not actually the printer that is making the gui freezes. It is the line that sets the html to the QTextDocument that is slowing down. Any suggestions?

    doc.setHtml(htmltable);
    

  • Moderators

    I guess you're doing the work twice. Instead of building html string and then parse it to convert to QTextDocument maybe you could just build the QTextDocument directly using QTextCursor. That should be faster.



  • Thanks for the suggestion but as i already have a worker thread used for stg else, i am now printing in the worker thread and im sending the string via a signal.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.