QNetworkAccessManager - first GET very slow

Aerius

I have a problem with using the QNetworkAccessManager in Qt 5.5 and Qt 5.6 on android, ios and mac osx. Downloading a simple, small graphic file via http GET results in a lot of waiting timeand a lockup of the UI during that time. Subsequent GETs work flawlessly and fast.

void DownloadManager::downloadFile(QUrl fromUrl, QString toFilePath) {

    _currentFilePath = toFilePath;

    QNetworkRequest request;
    request.setUrl(fromUrl);

    qDebug() << "before";

    _currentReply = _mgr.get(request);

    qDebug() << "after";

    connect(_currentReply, SIGNAL(error(QNetworkReply::NetworkError)), this, SLOT(downloadError(QNetworkReply::NetworkError)));
    connect(_currentReply, SIGNAL(downloadProgress(qint64, qint64)), this, SLOT(downloadProgress(qint64,qint64)));
    connect(_currentReply, SIGNAL(finished()), this, SLOT(downloadFinished()));

}

DownloadManager is a custom QObject-derived class without any special features that are relevant to the get request. _mgr is a QNetworkAccessManager Object that's allocated during DownloadManagers cTor.

As you can see, this is just a textbook example of a get request, nothing too fancy about it. And as I said: it works, for the most part. Only the first get request is very slow. On Android, the application output gives me a lot of Garbage Collection calls:

D/ .../downloadmanager.cpp:61 (void DownloadManager::downloadFile(QUrl, QString)): before
D/dalvikvm(13298): GC_CONCURRENT freed 2290K, 25% free 10911K/14407K, paused 2ms+3ms, total 29ms
D/dalvikvm(13298): GC_CONCURRENT freed 501K, 25% free 10884K/14407K, paused 13ms+2ms, total 35ms
D/dalvikvm(13298): GC_CONCURRENT freed 524K, 25% free 10892K/14407K, paused 12ms+3ms, total 36ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 6ms
D/dalvikvm(13298): GC_CONCURRENT freed 537K, 25% free 10887K/14407K, paused 2ms+9ms, total 32ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 14ms
D/dalvikvm(13298): GC_CONCURRENT freed 840K, 25% free 10899K/14407K, paused 12ms+3ms, total 38ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 11ms
D/dalvikvm(13298): GC_CONCURRENT freed 1294K, 25% free 10901K/14407K, paused 2ms+2ms, total 27ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 11ms
D/dalvikvm(13298): GC_CONCURRENT freed 1187K, 22% free 11330K/14407K, paused 2ms+2ms, total 30ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 15ms
D/dalvikvm(13298): GC_CONCURRENT freed 1459K, 19% free 11919K/14535K, paused 13ms+4ms, total 64ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 18ms
D/ .../downloadmanager.cpp:65 (void DownloadManager::downloadFile(QUrl, QString)): after

On the other platforms, I don't get this info, but that's probably just the case because it simply doesnt show up. The lockup and time waiting is still there (albeit much lower, though this could very much be a case of performance of the respective device, less of the platform overall).

Additional Information:

It is always only the first download that triggers this. Subsequent downloads, even for the exact same file, work flawlessly
It doesn't matter if there's a file at the exact location with the exact name or not. Downloading the file, deleting it, getting back into the application and redownloading it provides the same results - the first get is slow and has the GC, the second works perfectly fine.
I call all that from a QML File, that causes a singleton c++ object to call DownloadManager::downloadFile. I doubt this matters, as the c++ code is very much working on it's own.
Other than the QML UI, nothing else is running within the application. No heavy data exchanges, no background loading on other threads, nothing.
As I said, it happens on all platforms I tested it on, the most notable platform is android because of the visible GC calls.

I'd be thankful for any pointers towards solving this.

SGaist

Hi,

Does it happen independently of which server you are sending the request to ?

Aerius

Hello SGaist,

yes, it doesn't matter which Server I download from. Even if I download from multiple servers with several get requests, it's always the first server that ends up being slow (even if I just change the order of get requests to the same servers).

SGaist

Do you have any proxy or firewall in your network setup ?

Aerius

No, none that I'm aware of. Regular home network without special routing and/or protection. I might get around checking it in a different network, but regardless of the result, locking up the UI for a second for a GET request is not exactely great. I'm fine with waiting longer if it really is a pure networking issue, but the whole UI lockup is just not something I'm willing to accept if I can avoid it.

SGaist

GUI freeze ? On all the platforms you mentioned ?

Aerius

Yes, as I mentioned in the first post, during the slow GET, the gui is completely frozen up, waiting for it to finish. On all platforms. During that time on android, the application throws those concurrent GC 'warnings'.

SGaist

Can you share a sample request ? i.e. the URL that you are trying to get ?

kshegunov

@Aerius
What about DNS lookups? Could you try retrieving a file directly with a server's IP, so not to cause the whole DNS fetching nonsense to happen?

locking up the UI for a second for a GET request is not exactely great.

Unless QML uses some threading internally, which I don't believe to be the case, your whole network request is executed in the main thread. Fetching any data through the main thread could potentially cause the UI to freeze, so I don't see a reason to be surprised.

One additional remark:

I call all that from a QML File, that causes a singleton c++ object to call DownloadManager::downloadFile. I doubt this matters, as the c++ code is very much working on it's own.

You are not supposed to create QObject instances before the root QObject is alive (application object, QML engine or however the QML runtime is being set up), so depending on your (singleton) implementation you may be getting weird results from that as well.

Aerius

@SGaist said:

Can you share a sample request ? i.e. the URL that you are trying to get ?

https://forum.qt.io/logo.png (qt logo)
http://spiegel.de/static/sys/v10/logo/spiegel_online_logo_460_64.png (logo of a large german news website)
http://mediadb.kicker.de/library/image/logo-kicker.png (logo of a german football website)

I tried several others from several other sources and servers to no difference. And again, for instance, if I take these three files, whatever file is the first one to get downloaded triggers the lockup, subsequent ones do not. It doesn't matter if I download the qt logo first, for instance, or the kicker one. Whatever's the first in line locks up everything.

@kshegunov said:

@Aerius
What about DNS lookups? Could you try retrieving a file directly with a server's IP, so not to cause the whole DNS fetching nonsense to happen?

I'm not sure about how I would go about that, it's not like I can just replace the URL with an IP string. Willing to do so, but I'd need a pointer on where to look that up.

Unless QML uses some threading internally, which I don't believe to be the case, your whole network request is executed in the main thread. Fetching any data through the main thread could potentially cause the UI to freeze, so I don't see a reason to be surprised.

The thing is, QNAM explicitely states that it handles it's own requests asynchronously on other threads - and from what I can see, this is very much the case for every request but the first one . or alternatively, the first one triggers something that causes the lockup. Am I mistaken in my assumption here?

One additional remark:

I call all that from a QML File, that causes a singleton c++ object to call DownloadManager::downloadFile. I doubt this matters, as the c++ code is very much working on it's own.

You are not supposed to create QObject instances before the root QObject is alive (application object, QML engine or however the QML runtime is being set up), so depending on your (singleton) implementation you may be getting weird results from that as well.

The qml engine creates the c++ singleton object, it's registered via qmlRegisterSingletonType<>(). I do not programatically create any c++ objects before the one singleton object (which holds my entire c++ stuff).

On an important sidenote, even if the QNetworkReply::error() signal is emitted, the lockup/GC calls still happen before it's emitted. I'm pretty sure that the lockup doesn't have anything to do with the download itself, but happens as soon as QNetworkAccessManager starts any kind of get request for the very first time, regardless of what it tries to download.

Also: Thanks for your answers and thoughts so far. Communication over different timezones is a little difficult, but I'm happy about any help I can get here.

kshegunov

@Aerius
Hello,

I'm not sure about how I would go about that, it's not like I can just replace the URL with an IP string. Willing to do so, but I'd need a pointer on where to look that up.

Only replace the server name with the IP. If there is no special scheme to serving the requests this should be sufficient.

For example, the google logo:
http://www.google.bg/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png
On my machine google.bg resolves to 216.58.212.35, so using that IP I can access the image as well:
http://216.58.212.35/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png

The thing is, QNAM explicitely states that it handles it's own requests asynchronously on other threads - and from what I can see, this is very much the case for every request but the first one.

I admit, I don't know the documentation by heart, but where does it state that?

The qml engine creates the c++ singleton object, it's registered via qmlRegisterSingletonType<>(). I do not programatically create any c++ objects before the one singleton object (which holds my entire c++ stuff).

Fair enough. Then it's tied to and is handled by the QML engine so my guess is you're just fine with it.

Aerius

@kshegunov said:

@Aerius
Hello,

I'm not sure about how I would go about that, it's not like I can just replace the URL with an IP string. Willing to do so, but I'd need a pointer on where to look that up.

Only replace the server name with the IP. If there is no special scheme to serving the requests this should be sufficient.

For example, the google logo:
http://www.google.bg/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png
On my machine google.bg resolves to 216.58.212.35, so using that IP I can access the image as well:
http://216.58.212.35/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png

Interesting. For google this works for me too, for other servers it did not. Might be a server configuration thing? Not my area of expertise, if you couldn't guess that by now.
In any case - resolving the url to an ip adress in the get request doesn't change anything with my problem - still the same issue, still only for the first get.

The thing is, QNAM explicitely states that it handles it's own requests asynchronously on other threads - and from what I can see, this is very much the case for every request but the first one.

I admit, I don't know the documentation by heart, but where does it state that?

http://doc.qt.io/qt-5/qnetworkaccessmanager.html

QNetworkAccessManager has an asynchronous API. When the replyFinished slot above is called, the parameter it takes is the QNetworkReply object containing the downloaded data as well as meta-data (headers, etc.).

Moreover - later get requests work fine asynchronously. It's always only the first one that locks up the ui (which leads me to believe it has nothing to do with the actual download, as I stated before).

kshegunov

QNetworkAccessManager has an asynchronous API.

This doesn't necessarily mean that the code is threaded. In any case I got curious and opened the source to see what's going on. So yes, I can confirm that a thread is started, but for asynchronous request a network access manager global thread is used, meaning all replies are processed in one thread. This is one possibility as to why your first request is slower than the others.

In the request creation/reply setup I don't see anything special for android except an "assets" protocol. One thing that looks that might be locking the main thread is the backend and session initialization as they seem to be performed in the main thread, if I'm reading the source correctly.

Another thing I saw is that a proxy lookup is done in the main thread and while you're not using a proxy it might also take some time to refresh/resolve or possibly the caching is starting up a slower than expected.

So you see there are multiple possible reasons and I suggest investigating each one of them separately by fiddling with the access manager's properties a bit. For example you could try disabling/forcing caching. Or trying to trace where in the Qt source the delay could be occurring (you'd want to build Qt from source for that).
Although it's not really a direct answer to your problem, I hope this helps.

Kind regards.

djee

This is a bit old, but one of the first results on Google and neither the actual reason nor the workaround seem to be explained, so here they are: on first get(), the network manager will lazily load some library (DLL), which can take quite some time and freeze the UI if the code expects (rightly) that get() is purely asynchronous and will return quickly. The workaround is to pre-connect to the given server with connectToHost() or connectToHostEncrypted() earlier. After doing so the first get() has the same performance than subsequent ones.

harry159821

@djee In My Way connectToHost To Http Server Not Working, So I Call QSslConfiguration::defaultConfiguration(); Instead.