Lagging window/gui when using realtime table view

Mr Gisa

I have a class that requests a website, that website returns a json that is paginated, each page has a lot of urls. Each time that class of mine gets a new url from that json it emits a signal called handleUrl(const QUrl &url), that url is requested again in another method and with the results it generates a new Result and a new signal is emitted resultAvailable(Result *result).
I have a model with a method that is called addResult(Result *result), every time the signal resultAvailable is emitted it will call the addResult and add a new result to the model and consequently to the table view.
The problem is that as it's in real time, as it's going to get a new url, request, create a new result and insert in the model it lags the gui, every time a new result is added to the table it lags the ui, when trying to move the window it feels like jumping and note actually moving, also the hovering effect of the buttons is really slow.
How to improve the performance of a real time adding of items in the table?

JKSH

@Mr-Gisa It's a bit hard to tell what's causing the lag from your descriptions alone. However, it sounds like you have a function (or several) that run for a long time before it returns. It probably has nothing to do with the Table View itself.

Some questions:

How do you get data from the website?
Which function(s) in your program takes the longest to run? (One way to find out is to profile your code)
How frequently (i.e. how many times a second) does your signal get emitted?
Are you using any Qt::DirectConnection anywhere in your program?

Mr Gisa

1 - In a class I send a request to a page, that page returns me a json with a few information, each item of that json contains an url that needs to be requested later.
That class paginates the requests, for example, it will request the first page, grab the json, parse the urls, and try to call the next page, if the next page doesn't have results it's the end of pagination.
For each url found in each json returned, the class emits a signal urlAvailable.
Now another class that listen to this signal creates a new request for that url async, as QNetworkAccessManager handles 6 requests async it will continue to receive urls and continue to creating requests.
After each page has been requested the software will parse the html of that page using regex, it will also instantiate a new Result and put the information in there, that same method will also trigger resultAvailable, that signal is connected to the addResult of the model.

request page > grab json > parse json (for each url -> emit urlAvailable) > get next page (until there's no results anymore)

2 - I have no idea as most of the functions are being called by signals & slots, I don't call the methods manually. it's like one thing connected to the other until the final signal is emitted to the model in order to add the Result.

3 - A lot, I mean, as it's concurrent the requests, it depends on the return of the QNetworkAccessManager and the parsing of the html content using regex.

urlAvailable > request url > parse content with regex (create and emit resultAvailable)

4 - I'm not.

Just as an additional to what you said about a method that is taking too long to return, I don't know about that because the result is only added to the model when it's available. You think that some method is blocking the gui? If yes, why it doesn't block when I add the result to the tableview? It works normally if I just print the Result information instead of showing?
If some method is blocking the gui, will I have to put this in another thread of there's a solution for that?

JKSH

@Mr-Gisa said in Lagging window/gui when using realtime table view:

You think that some method is blocking the gui?

Yes, based on your descriptions in your original post.

It might not be a single a method. A very long chain of signals and slots has the same effect: Within the same thread, the function that emitted the first signal is blocked until until the last slot in the chain returns.

If yes, why it doesn't block when I add the result to the tableview? It works normally if I just print the Result information instead of showing?

This line is difficult to understand. Do you mean "add the result to the model"? You can't add data to the view, after all. Also, please be more specific: How do you "print"? How do you "show"?

I'm guessing you meant: The program lags when resultAvailable() is emitted (which runs addResult()). In contrast, it doesn't lag if you simply print the Result object's data to the console (using printf() or std::cout or qDebug()) without emitting the signal.

Am I on the right track? If so, then addResult() might contain inefficient code.

Quick check: What happens if you don't link your model to the table view (comment out tableView->setModel(...)? Does the UI still lag?

If some method is blocking the gui, will I have to put this in another thread of there's a solution for that?

If the block is caused by the network requests or the parsing, then you can move it to another thread.

If the block is caused by inefficient code in your model, then you probably cannot move it to another thread. You'll have to optimize your model's code within the GUI thread.

If the block is caused by inefficient code in Qt, then you'll need to get the problem fixed upstream.

I have no idea as most of the functions are being called by signals & slots, I don't call the methods manually.

Regardless of whether your function is called manually or triggered by a signal, knowing how to profile your code is a useful skill. I recommend you spend some time to learn it. Here's one quick example, but there are more sophisticated tools out there: https://www.wikihow.com/Optimize-Your-Program's-Performance

Profiling helps you identify the cause of your lag. Without it, you'll have to resort to trial-and-error.

Mr Gisa

@JKSH said in Lagging window/gui when using realtime table view:

I'm guessing you meant: The program lags when resultAvailable() is emitted (which runs addResult()). In contrast, it doesn't lag if you simply print the Result object's data to the console (using printf() or std::cout or qDebug()) without emitting the signal.

Exactly, sorry for the wrong description.

Quick check: What happens if you don't link your model to the table view (comment out tableView->setModel(...)? Does the UI still lag?

It doesn't lag at all.

That is the entire model code:

#include "resultsmodel.hpp"
#include "result.hpp"

ResultsModel::ResultsModel(QObject *parent) : QAbstractTableModel(parent)
{
    // ...
}

int ResultsModel::rowCount(const QModelIndex &) const
{
    return mResults.size();
}

int ResultsModel::columnCount(const QModelIndex &) const
{
    return 5;
}

QVariant ResultsModel::data(const QModelIndex &index, int role) const
{
    if (! index.isValid()) {
        return QVariant();
    }

    if (index.row() >= mResults.size() || index.row() < 0) {
        return QVariant();
    }

    auto result = mResults.at(index.row());

    if (role == Qt::DisplayRole) {
        switch (index.column()) {
        case 0:
            return result->filename().isEmpty() ? "?" : result->filename();
            break;

        case 1:
            return result->url();
            break;

        case 2:
            return result->source();
            break;

        case 3:
            return result->fileSize().isEmpty() ? "?" : result->fileSize();
            break;

        case 4:
            return result->isOnline() ? "Yes" : "No";
            break;
        }
    }

    return QVariant();
}

bool ResultsModel::setData(const QModelIndex &, const QVariant &, int)
{
    return false;
}

QVariant ResultsModel::headerData(int section, Qt::Orientation orientation, int role) const
{
    if (role != Qt::DisplayRole) {
        return QVariant();
    }

    if (orientation == Qt::Horizontal) {
        switch (section) {
        case 0:
            return tr("Filename");
            break;

        case 1:
            return tr("Url");
            break;

        case 2:
            return tr("Source");
            break;

        case 3:
            return tr("Size");
            break;

        case 4:
            return tr("Online");
            break;

        default:
            return QVariant();
            break;
        }
    }

    return QVariant();
}

Qt::ItemFlags ResultsModel::flags(const QModelIndex &index) const
{
    if (! index.isValid()) {
        return Qt::ItemIsEnabled;
    }

    return QAbstractTableModel::flags(index);
}

void ResultsModel::addResult(Result *result)
{
    if (mResults.contains(result)) {
        return;
    }

    auto index = mResults.size();

    beginInsertRows(QModelIndex(), index, index);
    mResults << result;
    endInsertRows();
}

void ResultsModel::reset()
{
    beginResetModel();
    qDeleteAll(mResults.begin(), mResults.end());
    mResults.clear();
    endResetModel();
}

JKSH

@Mr-Gisa said in Lagging window/gui when using realtime table view:

@JKSH said in Lagging window/gui when using realtime table view:

I'm guessing you meant: The program lags when resultAvailable() is emitted (which runs addResult()). In contrast, it doesn't lag if you simply print the Result object's data to the console (using printf() or std::cout or qDebug()) without emitting the signal.

Exactly, sorry for the wrong description.

Quick check: What happens if you don't link your model to the table view (comment out tableView->setModel(...)? Does the UI still lag?

It doesn't lag at all.

That is the entire model code:

Your code looks quite straightforward and clean.

I'm wondering if beginInsertRows() is being called too frequently. Some things you can try:

Do some very basic profiling: Make every call to addResult() print a message to the console. Roughly how frequently is it being called?
Use batched updates. Instead of doing beginInsertRows() - append - endInsertRows() every single time the resultAvailable() signal is emitted, store the new result in a temporary location but don't add it to mResults yet. Every 0.5s, add all the new results at once:
1. Call beginInsertRows() once
2. Add all the available results
3. Call endInsertRows() once

Mr Gisa

@JKSH with batched results you mean like this? https://doc.qt.io/qt-5/qtwidgets-itemviews-fetchmore-example.html

About the how many times the addResult is being called, it's a lot, it's like every half second a new result is added.

Christian Ehrlicher

@Mr-Gisa said in Lagging window/gui when using realtime table view:

if (mResults.contains(result)) {
    return;
}

This is not a good idea...

why should addResult() called twice with the same pointer? This can only happen due to wrong code on your side
a linear lookup is maybe fast for < 100 lookups but not for huge data.

Christian Ehrlicher

@Mr-Gisa said in Lagging window/gui when using realtime table view:

if (! index.isValid()) {
    return QVariant();
}

if (index.row() >= mResults.size() || index.row() < 0) {
    return QVariant();
}

auto result = mResults.at(index.row());

And here I would first check for the role, then the bounds and at last for the validity since this is the most expensive operation here.

Mr Gisa

@Christian-Ehrlicher I made the changes you pointed out and it helped a lot, it's still lagging, but way less than it was before. I can feel the gui freezing when trying to move, and as it's a lot of incoming data per second in the model you can really feel the lag in the gui.
I think you know a software called Wireshark, it was made using Qt and their table receives A LOT (way more than mine) of packets information and it doesn't lag at all, but in my application I can feel the lag when I try to move the window around while the results are being displayed in the table.

Christian Ehrlicher

Another idea is to collect some results and add them once in a second (or every n seconds) instead every single one.

Mr Gisa

I realized something. When I compile in release mode and run the application out of the Qt, like running the actual executable it doesn't lag. Why is that?

Christian Ehrlicher

Because now the compiler don't need to generate debug information and compiles your code with optimizations.

Mr Gisa

You think I should leave that way getting results at real time or do the batched thing? What is the best alternative here?

mrjj

@Mr-Gisa
Since you are planning on even higher payloads, you should test if release mode
can handle that or you simply need to batch them.

Mr Gisa

It will have on average 100 or 200 thousand results.

mrjj

@Mr-Gisa
and are you testing with that many now?

Mr Gisa

5~10 thousands, I can't get to 100~200 cause I have to code the other APIs, but with that it's working normally at this point.

mrjj

@Mr-Gisa
Ok, but you can postpone it until you have test data then and work on the other parts.

Mr Gisa

Exactly, that is what I'm going to do. Thank you all for your help. If anything happen in the future I'll say it here.
I'm also going to get a copy with a friend of mine of his Intel Parallel Studio 2018 to check a few things in the application, I hope it works with Qt.