Unsolved QNetworkReply and setReadBufferSize problem
-
@Hergest said in QNetworkReply and setReadBufferSize problem:
I don't want to have it in memory, but what I download is not a file
doesn't really matter, download as fast as possible, store it in a
QTemporaryFile
and use theQTemporaryFile
asQIODevice
in input to your DB uploader instead of theQNetworkReply
. They use the same interface so it should be super easy to replace one for the other:QTemporaryFile* destination =new QTemporaryFile(networkAccessManager); if(destination->open(QFile::WriteOnly)){ QNetworkRequest request(QUrl::fromUserInput("https://zlib.net/zlib-1.2.11.tar.gz")); QNetworkReply *reply = networkAccessManager->get(request); QObject::connect(reply,&QNetworkReply::readyRead,destination,[=](){ destination->write(reply->readAll()); }); QObject::connect(reply,&QNetworkReply::finished,reply,&QNetworkReply::deleteLater); QObject::connect(reply,QOverload<QNetworkReply::NetworkError>::of(&QNetworkReply::error),destination,&QTemporaryFile::deleteLater); QObject::connect(reply,QOverload<QNetworkReply::NetworkError>::of(&QNetworkReply::error),reply,&QNetworkReply::deleteLater); }
-
So I have to do it this way, and setReadBufferSize doesn't really work?
The program will be multiplatform, and I don't know if that matches a mobile app.
Can anyone confirm to me that it doesn't work?Thanks!
-
@VRonin i wonder if you made a mistake or this is something i don't know yet
QObject::connect(reply,&QNetworkReply::readyRead,destination,[=](){destination->write(reply->readAll());});
Please tell me why do you pass 'destination' to 'connect()' in the reciver place ?
-
@Hergest said in QNetworkReply and setReadBufferSize problem:
Can anyone confirm to me that it doesn't work?
I can confirm
setReadBufferSize
does not "slow down" the downloadTest program:
#include <QDebug> #include <QNetworkRequest> #include <QNetworkAccessManager> #include <QNetworkReply> #include <QThread> int main(int argc, char **argv) { QCoreApplication app(argc,argv); QNetworkAccessManager networkAccessManager; QNetworkRequest request; request.setUrl(QUrl::fromUserInput("https://codeload.github.com/qt/qtbase/zip/5.11")); QNetworkReply *reply = networkAccessManager.get(request); reply->setReadBufferSize(1024); QObject::connect(reply,&QNetworkReply::readyRead,[=](){ QThread::sleep(3); qDebug() << "available: " << reply->bytesAvailable(); const QByteArray readData = reply->readAll(); qDebug() << "read " << readData.size() << "bytes"; }); QObject::connect(reply,&QNetworkReply::downloadProgress,[](qint64 bytesReceived, qint64 bytesTotal){ if(bytesTotal<=0) return; qDebug() << bytesReceived << '/' << bytesTotal; }); QObject::connect(reply,&QNetworkReply::finished,reply,&QNetworkReply::deleteLater); QObject::connect(reply,&QNetworkReply::finished,&app,&QCoreApplication::quit); QObject::connect(reply,QOverload<QNetworkReply::NetworkError>::of(&QNetworkReply::error),reply,&QNetworkReply::deleteLater); return app.exec(); }
and I don't know if that matches a mobile app
QSaveFile
is part of Qt core, it will work on all platforms -
@LeLev said in QNetworkReply and setReadBufferSize problem:
Please tell me why do you pass 'destination' to 'connect()' in the reciver place ?
It's not the receiver, it's the context. Basically it tells
connect
that ifdestination
is destroyed, that connection should be severed as well, otherwise you'd risk accessing a dangling pointer inside the lambda -
@VRonin last one before i stop flood this topic : so i don't need to do this when i call a method on member variable right ?
... private: SFtpClient m_sftp;// ... QObject::connect(connect_btn,&QPushButton::clicked,[=](){ m_sftp->connectToMachine("user@10.81.100.100"); });
or
private: SFtpClient m_sftp;// ... QObject::connect(connect_btn,&QPushButton::clicked,m_sftp,[=](){ m_sftp->connectToMachine("user@10.81.100.100"); });
-
@LeLev said in QNetworkReply and setReadBufferSize problem:
so i don't need to do this when i call a method on member variable right ?
Depends. If the signal emitter can't survive
this
(e.g.connect_btn
is a child ofthis
) then no, but if there's a chanceconnect_btn
can still emit a signal oncethis
was destroyed (and hencem_sftp
was destroyed as well) then you have to addthis
or&m_sftp
as 3rd argument toconnect
to make it safe -
QSaveFile is part of Qt core, it will work on all platforms
The problem is the size of file, not the compatibility.
I understand something's broken at Qt so it won't work. I can't find it in QTBUG. There are traces in version 4.8, but it says "Solved".
Perhaps a "bug request" should be opened... -
@Hergest said in QNetworkReply and setReadBufferSize problem:
Perhaps a "bug request" should be opened
You can do it
-
@Hergest said in QNetworkReply and setReadBufferSize problem:
Can anyone confirm to me that it doesn't work?
It works just fine, that is it sets the read buffer size, which has nothing to do with your problem. As @VRonin said: open a file and stream the network data into it. Whenever you have an opening in the event loop, e.g. you can have a timer or connect to the
bytesWritten
from theQIODevice
, read from the file and process its contents.I understand something's broken at Qt so it won't work.
What won't work? If you mean
setReadBufferSize
, then you're wrong, it works, just it does not do what you expect it to do.@LeLev said in QNetworkReply and setReadBufferSize problem:
last one before i stop flood this topic : so i don't need to do this when i call a method on member variable right ?
You don't need to as long as you understand that the sender is going to be used as context if you don't specify anything. A rule of thumb to avoid problems that are hard to diagnose is just to always provide the context explicitly.
-
@kshegunov said in QNetworkReply and setReadBufferSize problem:
What won't work? If you mean setReadBufferSize, then you're wrong, it works, just it does not do what you expect it to do.
Looking the help:
Sets the size of the read buffer to be size bytes. The read buffer is the buffer that holds data that is being downloaded off the network, before it is read with QIODevice::read(). Setting the buffer size to 0 will make the buffer unlimited in size.QNetworkReply will try to stop reading from the network once this buffer is full (i.e., bytesAvailable() returns size or more), thus causing the download to throttle down as well. If the buffer is not limited in size, QNetworkReply will try to download as fast as possible from the network.
What I expect is that it doesn't keep downloading data until the memory runs out, but that it stops until the buffer is emptied with read.
-
I'm pretty sure readBufferSize() is working as expected since it's an easy implementation: https://code.woboq.org/qt5/qtbase/src/network/access/qnetworkreplyimpl.cpp.html#181 (see nextDownstreamBlockSize() so you must do something wrong elsewhere but without code we can't do anything more.
-
@Hergest said in QNetworkReply and setReadBufferSize problem:
What I expect is that it doesn't keep downloading data until the memory runs out, but that it stops until the buffer is emptied with read.
It can try to throttle the download down if the server supports it, if it doesn't I don't see how this can happen. Consider the simplest scenario where the server just dumps the data over the network (which is what I imagine is happening) the networking stack has a limited buffer and (if set) Qt has a limited buffering, if you don't read the buffer fast enough then it is going to overflow. How do you handle the case where the protocol (or server) has no notion of speed? You can't just leave data pending in the buffers for when you're ready to process it.
-
This is a pseudocode of the function:
... netReply = m_pNetManager->post(req, qb_json); netReply->setReadBufferSize (64*1024); ... /* Reading some head data from netReply*/ ... size_blob = we get the size of the Blob to download from server ... /* Blob Reading from netReply // Loop while (...some conditions...){ ... Debug () << "bytesAvailable=" << netReply->bytesAvailable() << "bufferSize=" << netReply->readBufferSize(); qint64 bytes_read = 0; qint64 bytes_to_read = 1024*64; // chunk to download if ((bytes_read + bytes_to_read) > size_blob) bytes_to_read= size_blob - bytes_read ; qint64 parcial = netReply->read (reinterpret_cast<char*>(buffer_temporal), bytes_to_read); if (parcial > 0){ tmp_file.write (reinterpret_cast<const char*>(buffer_temporal), parcial); bytes_read += parcial; if (bytes_read >= size_blob){ // END READING status = 2; } } ... }
The real code reads the data sent by the server, first a header and then a blob (in 64KB chuncks), then another header and its blob, and so on.
In my tests, the value of "bytesAvailable" increases and surpasses "bufferSize", until the memory collapses.
When I'm debugging it, in a breakpoint, I check that the server stops sending data. As soon as I resume it, "bytesAvailable" continues to increase, and the server continues sending data (data from other blobs).
I have tried many combinations of chunks, ReadBufferSize values, ...This leads me to think...
- The server sends data whenever the client requests it. In the breakpoint it stopped doing it.
- Client keeps asking for data even though "readBuffer" fills up.
-
In the end I had to rewrite the code using QTcpSocket (which does obey the setReadBufferSize limitation), simulating POST requests.