QNetworkReply readyRead is not called, memory filling to max until crash

? Offline

@jsulm said in QNetworkReply readyRead is not called, memory filling to max until crash:

@ankou29666 Can you please tell us exactly (show the code) what you are doing with incoming data? You still refuse to tell us whether you're writing data directly to a file or accumulating it in RAM. If you accumulate all incoming data in RAM, then of course you will get out of memory if you're sending huge amount of data and this has nothing to do with Qt...
And the documentation you posted has nothing to do with what YOU are doing with incoming data.

I AM NOT THE ONE WITH THE PROBLEM
please be a little more careful about it.
just trying to help the way I can. I don't download gigabyte files (1kB file is big file in my case so far). All I'm saying is that I wonder what is the default buffer size, because when explicitly set to zero, the buffer size is limited only by the device's memory.

jsulm

@ankou29666 Sorry, I replied to you by mistake.

? Offline

@jsulm I had guessed. No worry this can happen to anyone ;)

robsparrow

Solved!
reply->setReadBufferSize(4096) did help, now readyRead signal is called and I can read data and release it from memory.

If here is anyone with power to do something with documentation then probably here https://doc.qt.io/qt-6/qnetworkreply.html#details should be added that readyRead event is directly controlled by setReadBufferSize(), because now documentation just says "Whenever more data is received from the network and processed, the readyRead() signal is emitted."

Also readyRead is not working as expected if we follow to this part of documentation - https://doc.qt.io/qt-6/qnetworkreply.html#setReadBufferSize
"Setting the buffer size to 0 will make the buffer unlimited in size." as default value is 0. I would not expect even 1 signal for readyRead, but I was getting from 1-7 signals even in first seconds while file downloading until memory was filled or app was not responsive. but file was not fully downloaded while those signals was sent. So why there is such random behaviour on this signal..

Christian Ehrlicher

@robsparrow said in QNetworkReply readyRead is not called, memory filling to max until crash:

So why there is such random behaviour on this signal..

There is no random behavior on this signal:

"This signal is emitted once every time new data is available for reading from the device's current read channel. It will only be emitted again once new data is available, such as when a new payload of network data has arrived on your network socket, or when a new block of data has been appended to your device."

Nothing about buffer sizes here

? Offline

@Christian-Ehrlicher said in QNetworkReply readyRead is not called, memory filling to max until crash:

Nothing about buffer sizes here

I think it might. When downloading short files, the signal will only be sent once the request is totally complete. I don't know how it works in internal but I wouldn't be surprised that readyRead signal would be fired only once the bytecount in the buffer matches the bytecount declared in the reply's http header.

I don't know if I'm right or wrong, but in that case it would explain why he never received the readyRead signal if he gets a memory overflow before the file is completely downloaded.

Christian Ehrlicher

@ankou29666 said in QNetworkReply readyRead is not called, memory filling to max until crash:

I think it might. When downloading short files, the signal will only be sent once the request is totally complete. I don't know how it works in internal but I wouldn't be surprised that readyRead signal would be fired only once the bytecount in the buffer matches the bytecount declared in the reply's http header.

No - it's clearly written: "This signal is emitted once every time new data is available for reading from the device's read channel"

? Offline

@Christian-Ehrlicher this statement is obviously wrong.

by the way what do they actually mean by "new data" ? and same about "availability".

Christian Ehrlicher

@ankou29666 said in QNetworkReply readyRead is not called, memory filling to max until crash:

this statement is obviously wrong.

Ok, no need to discuss further here...

Qt waits for an event from the OS and emits a queued signal readyRead() so it depends on the current OS + CPU usage...

ChrisW67

The OP's unmodified code, on my machine with just 16GB of RAM: retrieving a local file:

$ curl -I http://localhost/backup-2022.tar.gz
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 28 Feb 2023 07:53:57 GMT
Content-Type: application/octet-stream
Content-Length: 15072231424
Last-Modified: Tue, 28 Feb 2023 07:42:03 GMT
Connection: keep-alive
ETag: "63fdb04b-382600000"
Accept-Ranges: bytes

Emits readyRead thousands of times, reads, and discards the data as expected and documented.
Finishes cleanly
Barely cracks 1% of RAM used
Linux Qt 6.4.2 and 5.15.2

Exhausting RAM without completing is environmental, a Windows platform issue, or web server oddity (is it CPU bound for example).

The only time I had a lone readyRead() call was the web server returning an HTTP 403 (PEBKAC).

Initially, downloading multiple times in the same run consumed an extra memory chunk each time because the QNetworkReply is not deleted. Fixing that fixes this growth.

JonB

@robsparrow
The results shown by @ChrisW67 under Linux are indeed what would be expected.

One thing to check: your earlier Memory usage screenshot from Task Manager > Performance looks like it's showing for whole machine? Can you verify whether that memory is being used by your Qt process versus whether it is attributed to something else, like the OS itself?

? Offline

@JonB that wasn't my graphs !!! All I said is I was wondering what the behavior is for big files. Personally I'm not concerned by such file sizes, and haven't yet reached the point where I will have to deal with bigger file sizes (and I'll still be well below RAM quantity anyways).

I have already seen inaccurate documentations ;) so sometimes I take them a little cautiously.

JonB

@ankou29666 My fault, I meant to address that to the OP @robsparrow, corrected.

J.Hilk

@JonB its the new forum style at fault here,

@ankou29666 and @robsparrow have the same color back ground image.

It's also the only thing I tend to look at, at first glance at least

JonB

@J-Hilk
I also find the new behaviour here when typing @... since the forum update confusing.... Though it's probably not to blame for my fault above :)

J.Hilk

@JonB In think that got reverted!

JonB

@J-Hilk
Umm, nope, try typing just @ versus @ followed by something. And it seems to behave differently inside a post, at least when I come back to edit one I have already made.... However this is now OT in this unsolved thread....

robsparrow

Windows 10 Build 19044. CPU - Ryzen 9 4900H.

Tests:

without "setReadBufferSize()" - each app rerun gives different count of readyRead signals until app starts filling memory until crash - usually 0-3 signals in first 2-5 seconds and then its not responding anymore and memory is filled until crash
with reply->setReadBufferSize(1024); - readyRead is called many times per second, memory is not filled, everything is fine

Its not making any load on CPU even in test case (1) when memory is filled.

? Offline

@JonB said in QNetworkReply readyRead is not called, memory filling to max until crash:

@ankou29666 My fault, I meant to address that to the OP @robsparrow, corrected.

no worry, not the first time this happens in this topic (but as this topic is tagged as solved, i think I can reasonably hope this is the last !!!)

@robsparrow said in QNetworkReply readyRead is not called, memory filling to max until crash:

Windows 10 Build 19044. CPU - Ryzen 9 4900H.

Tests:

without "setReadBufferSize()" - each app rerun gives different count of readyRead signals until app starts filling memory until crash - usually 0-3 signals in first 2-5 seconds and then its not responding anymore and memory is filled until crash

with reply->setReadBufferSize(1024); - readyRead is called many times per second, memory is not filled, everything is fine

Its not making any load on CPU even in test case (1) when memory is filled.

thanks for the share, that's good to know

JoeCFD

@robsparrow has the buffer size to be 1024? Have you tried any other numbers for comparison?