QNetworkReply and setReadBufferSize problem



  • Hello,
    I'm making a download application using QNetworkAccessManager and QNetworkReply.
    After a POST request, the server streams back a large amount of data (a gigabyte or more).
    I have made a loop that reads and processes it from the QNetworkReply object, but the process memory increases until the application crashes with an out of memory error (the download is faster than the process).
    I've set the read buffer using setReadBufferSize, but it seems to be ignored, and everything that arrives is read, consuming all the memory.
    Can someone help me?

    Thanks in advance.


  • Qt Champions 2018

    @Hergest said in QNetworkReply and setReadBufferSize problem:

    (a gigabyte or more)

    Do you really need to keep that much data in memory?!

    The normal process is to redirect the downloaded data directly to a file:

    QSaveFile* destination =new QSaveFile(QStringLiteral("downloaded.tar.gz"),networkAccessManager);
    if(destination->open(QFile::WriteOnly)){
    QNetworkRequest request;
    request.setUrl(QUrl::fromUserInput("https://zlib.net/zlib-1.2.11.tar.gz"));
    QNetworkReply *reply = networkAccessManager->get(request);
    QObject::connect(reply,&QNetworkReply::readyRead,destination,[=](){destination->write(reply->readAll());});
    QObject::connect(reply,&QNetworkReply::finished,destination,&QSaveFile::commit);
    QObject::connect(reply,&QNetworkReply::finished,destination,&QSaveFile::deleteLater);
    QObject::connect(reply,&QNetworkReply::finished,reply,&QNetworkReply::deleteLater);
    QObject::connect(reply,QOverload<QNetworkReply::NetworkError>::of(&QNetworkReply::error),destination,&QSaveFile::deleteLater);
    QObject::connect(reply,QOverload<QNetworkReply::NetworkError>::of(&QNetworkReply::error),reply,&QNetworkReply::deleteLater);
    }
    


  • No, I don't want to have it in memory, but what I download is not a file, but data that I process and insert in DB.
    The problem is that the download is faster than my processes until I put it in the DB.
    Why doesn't setReadBufferSize work? Is it necessary to do more than set a value?


  • Qt Champions 2018

    @Hergest said in QNetworkReply and setReadBufferSize problem:

    I don't want to have it in memory, but what I download is not a file

    doesn't really matter, download as fast as possible, store it in a QTemporaryFile and use the QTemporaryFile as QIODevice in input to your DB uploader instead of the QNetworkReply. They use the same interface so it should be super easy to replace one for the other:

    QTemporaryFile* destination =new QTemporaryFile(networkAccessManager);
    if(destination->open(QFile::WriteOnly)){
        QNetworkRequest request(QUrl::fromUserInput("https://zlib.net/zlib-1.2.11.tar.gz"));
        QNetworkReply *reply = networkAccessManager->get(request);
        QObject::connect(reply,&QNetworkReply::readyRead,destination,[=](){
            destination->write(reply->readAll());
        });
        QObject::connect(reply,&QNetworkReply::finished,reply,&QNetworkReply::deleteLater);
        QObject::connect(reply,QOverload<QNetworkReply::NetworkError>::of(&QNetworkReply::error),destination,&QTemporaryFile::deleteLater);
        QObject::connect(reply,QOverload<QNetworkReply::NetworkError>::of(&QNetworkReply::error),reply,&QNetworkReply::deleteLater);
    }
    


  • So I have to do it this way, and setReadBufferSize doesn't really work?
    The program will be multiplatform, and I don't know if that matches a mobile app.
    Can anyone confirm to me that it doesn't work?

    Thanks!



  • @VRonin i wonder if you made a mistake or this is something i don't know yet

    QObject::connect(reply,&QNetworkReply::readyRead,destination,[=](){destination->write(reply->readAll());});
    

    Please tell me why do you pass 'destination' to 'connect()' in the reciver place ?


  • Qt Champions 2018

    @Hergest said in QNetworkReply and setReadBufferSize problem:

    Can anyone confirm to me that it doesn't work?

    I can confirm setReadBufferSize does not "slow down" the download

    Test program:

    #include <QDebug>
    #include <QNetworkRequest>
    #include <QNetworkAccessManager>
    #include <QNetworkReply>
    #include <QThread>
    
    int main(int argc, char **argv) {
        QCoreApplication app(argc,argv);
        QNetworkAccessManager networkAccessManager;
        QNetworkRequest request;
        request.setUrl(QUrl::fromUserInput("https://codeload.github.com/qt/qtbase/zip/5.11"));
        QNetworkReply *reply = networkAccessManager.get(request);
        reply->setReadBufferSize(1024);
        QObject::connect(reply,&QNetworkReply::readyRead,[=](){
            QThread::sleep(3);
            qDebug() << "available: " << reply->bytesAvailable();
            const QByteArray readData = reply->readAll();
            qDebug() << "read " << readData.size() << "bytes";
        });
        QObject::connect(reply,&QNetworkReply::downloadProgress,[](qint64 bytesReceived, qint64 bytesTotal){
            if(bytesTotal<=0)
                return;
            qDebug() << bytesReceived << '/' << bytesTotal;
        });
        QObject::connect(reply,&QNetworkReply::finished,reply,&QNetworkReply::deleteLater);
        QObject::connect(reply,&QNetworkReply::finished,&app,&QCoreApplication::quit);
        QObject::connect(reply,QOverload<QNetworkReply::NetworkError>::of(&QNetworkReply::error),reply,&QNetworkReply::deleteLater);
        return app.exec();
    }
    

    and I don't know if that matches a mobile app

    QSaveFile is part of Qt core, it will work on all platforms


  • Qt Champions 2018

    @LeLev said in QNetworkReply and setReadBufferSize problem:

    Please tell me why do you pass 'destination' to 'connect()' in the reciver place ?

    It's not the receiver, it's the context. Basically it tells connect that if destination is destroyed, that connection should be severed as well, otherwise you'd risk accessing a dangling pointer inside the lambda



  • @VRonin last one before i stop flood this topic : so i don't need to do this when i call a method on member variable right ?

    ...
    private:
    SFtpClient m_sftp;// 
    ...
    QObject::connect(connect_btn,&QPushButton::clicked,[=](){
            m_sftp->connectToMachine("user@10.81.100.100");
          });
    
    

    or

    private:
    SFtpClient m_sftp;// 
    ...
    QObject::connect(connect_btn,&QPushButton::clicked,m_sftp,[=](){
            m_sftp->connectToMachine("user@10.81.100.100");
          });
    
    

  • Qt Champions 2018

    @LeLev said in QNetworkReply and setReadBufferSize problem:

    so i don't need to do this when i call a method on member variable right ?

    Depends. If the signal emitter can't survive this (e.g. connect_btn is a child of this) then no, but if there's a chance connect_btn can still emit a signal once this was destroyed (and hence m_sftp was destroyed as well) then you have to add this or &m_sftp as 3rd argument to connect to make it safe



  • QSaveFile is part of Qt core, it will work on all platforms
    

    The problem is the size of file, not the compatibility.

    I understand something's broken at Qt so it won't work. I can't find it in QTBUG. There are traces in version 4.8, but it says "Solved".
    Perhaps a "bug request" should be opened...


  • Qt Champions 2018

    @Hergest said in QNetworkReply and setReadBufferSize problem:

    Perhaps a "bug request" should be opened

    You can do it


  • Qt Champions 2017

    @Hergest said in QNetworkReply and setReadBufferSize problem:

    Can anyone confirm to me that it doesn't work?

    It works just fine, that is it sets the read buffer size, which has nothing to do with your problem. As @VRonin said: open a file and stream the network data into it. Whenever you have an opening in the event loop, e.g. you can have a timer or connect to the bytesWritten from the QIODevice, read from the file and process its contents.

    I understand something's broken at Qt so it won't work.

    What won't work? If you mean setReadBufferSize, then you're wrong, it works, just it does not do what you expect it to do.

    @LeLev said in QNetworkReply and setReadBufferSize problem:

    last one before i stop flood this topic : so i don't need to do this when i call a method on member variable right ?

    You don't need to as long as you understand that the sender is going to be used as context if you don't specify anything. A rule of thumb to avoid problems that are hard to diagnose is just to always provide the context explicitly.



  • @kshegunov said in QNetworkReply and setReadBufferSize problem:

    What won't work? If you mean setReadBufferSize, then you're wrong, it works, just it does not do what you expect it to do.

    Looking the help:
    Sets the size of the read buffer to be size bytes. The read buffer is the buffer that holds data that is being downloaded off the network, before it is read with QIODevice::read(). Setting the buffer size to 0 will make the buffer unlimited in size.

    QNetworkReply will try to stop reading from the network once this buffer is full (i.e., bytesAvailable() returns size or more), thus causing the download to throttle down as well. If the buffer is not limited in size, QNetworkReply will try to download as fast as possible from the network.

    What I expect is that it doesn't keep downloading data until the memory runs out, but that it stops until the buffer is emptied with read.


  • Qt Champions 2018

    I'm pretty sure readBufferSize() is working as expected since it's an easy implementation: https://code.woboq.org/qt5/qtbase/src/network/access/qnetworkreplyimpl.cpp.html#181 (see nextDownstreamBlockSize() so you must do something wrong elsewhere but without code we can't do anything more.


  • Qt Champions 2017

    @Hergest said in QNetworkReply and setReadBufferSize problem:

    What I expect is that it doesn't keep downloading data until the memory runs out, but that it stops until the buffer is emptied with read.

    It can try to throttle the download down if the server supports it, if it doesn't I don't see how this can happen. Consider the simplest scenario where the server just dumps the data over the network (which is what I imagine is happening) the networking stack has a limited buffer and (if set) Qt has a limited buffering, if you don't read the buffer fast enough then it is going to overflow. How do you handle the case where the protocol (or server) has no notion of speed? You can't just leave data pending in the buffers for when you're ready to process it.



  • This is a pseudocode of the function:

    ...
    netReply = m_pNetManager->post(req, qb_json);
    netReply->setReadBufferSize (64*1024);
    ...
    /* Reading some head data from netReply*/
    ...
    size_blob =  we get the size of the Blob to download from server
    ...
    /* Blob Reading from netReply
    // Loop
    while (...some conditions...){
          ...
          Debug () << "bytesAvailable=" << netReply->bytesAvailable() << "bufferSize=" << netReply->readBufferSize();
          qint64 bytes_read = 0;
          qint64 bytes_to_read = 1024*64; // chunk to download
          if ((bytes_read + bytes_to_read) > size_blob)
            bytes_to_read= size_blob - bytes_read ;
          qint64 parcial = netReply->read (reinterpret_cast<char*>(buffer_temporal), bytes_to_read);
          if (parcial > 0){
            tmp_file.write (reinterpret_cast<const char*>(buffer_temporal), parcial);
            bytes_read += parcial;
            if (bytes_read >= size_blob){
              // END READING
              status = 2;
            }
          }
    ...
    }
    

    The real code reads the data sent by the server, first a header and then a blob (in 64KB chuncks), then another header and its blob, and so on.
    In my tests, the value of "bytesAvailable" increases and surpasses "bufferSize", until the memory collapses.
    When I'm debugging it, in a breakpoint, I check that the server stops sending data. As soon as I resume it, "bytesAvailable" continues to increase, and the server continues sending data (data from other blobs).
    I have tried many combinations of chunks, ReadBufferSize values, ...

    This leads me to think...

    1. The server sends data whenever the client requests it. In the breakpoint it stopped doing it.
    2. Client keeps asking for data even though "readBuffer" fills up.


  • In the end I had to rewrite the code using QTcpSocket (which does obey the setReadBufferSize limitation), simulating POST requests.


Log in to reply