Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. QNetworkAccessManager doesn't post request in case of large file
Forum Updated to NodeBB v4.3 + New Features

QNetworkAccessManager doesn't post request in case of large file

Scheduled Pinned Locked Moved Unsolved General and Desktop
26 Posts 6 Posters 4.0k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • R Offline
    R Offline
    Robert Winke
    wrote on last edited by Robert Winke
    #1

    Hello everyone,
    I'm trying to make a post request for a larger (~6Mb) text file with QNetworkAccessManager. The reply is connected to a lambda which should print out the progress however after very little (0.1%) progress I get Connection closed error (error code: 2). I'm monitoring the network traffic with wireshark and I don't even see the post request there

    HOWEVER if I use a small txt file then everything works fine (and I can see the request from the monitoring tool).

    So the only difference between the working and non-working scenario is the file size.

    Here is the relevant code snippet:

    void Gen2FirmwareUpdater::_UploadFWImage()
    {
        QString url = _restUrl.arg(_ipAddress).arg("upload");
    
        std::cout << "Uploading FW file to: " + url.toStdString() + "\n";
        QNetworkRequest postFileRequest(url);
    
        //postFileReqauest.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");
        postFileRequest.setHeader(QNetworkRequest::ContentTypeHeader, "text/plain");
        postFileRequest.setHeader(QNetworkRequest::ContentLengthHeader, _file->size());
    
        if (!_file->open(QIODevice::ReadOnly)) {
            std::cout << "Unable to open file\n ";
        }
    
        _pReply = _netMan.post(postFileRequest, _file.get());
        _file.release()->setParent(_pReply);   // now the reply will take care of
                                               // deleting the file handle
    
        auto onProgUpdate =
        [this](auto sent, auto total){
            if (total != 0) {
                auto prog = sent*100/(2*static_cast<double>(total));
                std::cout << "prog: " << prog << "\n";
            }
        };
    
        // Queue up the progress update so it doesn't intervene
        // with the underlying process
        QObject::connect(_pReply, &QNetworkReply::uploadProgress,
                         _pReply, onProgUpdate, Qt::QueuedConnection);
    
        QObject::connect(_pReply, &QNetworkReply::finished,
            [this](){
                auto error = this->_pReply->error();
                if (error) {
                    auto msg =
                    "Error Code: "   + std::to_string(static_cast<int>(error)) + "\n"
                    "\nError String: " + this->_pReply->errorString().toStdString();
                    std::cout << msg << "\n";
                } else {
                    std::cout << "New FW uploaded\n";
                }
                auto r = this->_pReply->readAll();
                if (r.size() > 0) {
                    std::cout << "Server response: " + r.toStdString() + "\n\n";
                }
            });
    
        //QEventLoop loop;
        //QObject::connect(_pReply, &QNetworkReply::finished, &loop, &QEventLoop::quit);
        //loop.exec();
    }
    

    I suspected that maybe something gets deleted before the upload completes so I'm using member variables for the network access manager and the reply (which should take ownership of the file).

    I have a python script which can do this upload so that eliminates server side problems.

    I also tried using QHttpMultiPart (not even sure that that is meant for this) but I had the same results.

    Here is a sandbox project with a working web server and readme on how to use it.
    https://drive.google.com/drive/folders/1lxPun_p_crKlui5KU8Q1qKBp0ue7vvok?usp=sharing
    (using this web server I receive Error Code: 99 Unknown error in case of large file but I can see the post reqest from wireshark, small file version still works perfectly)

    I hope there is someone to help me, thanks!

    1 Reply Last reply
    0
    • Christian EhrlicherC Offline
      Christian EhrlicherC Offline
      Christian Ehrlicher
      Lifetime Qt Champion
      wrote on last edited by
      #2

      What is _file? A shared pointer? Why?`And if - you know what release() does?

      Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
      Visit the Qt Academy at https://academy.qt.io/catalog

      R 1 Reply Last reply
      0
      • Christian EhrlicherC Christian Ehrlicher

        What is _file? A shared pointer? Why?`And if - you know what release() does?

        R Offline
        R Offline
        Robert Winke
        wrote on last edited by
        #3

        @Christian-Ehrlicher it's a unique pointer but I have also tried it with a raw pointer

        Christian EhrlicherC 1 Reply Last reply
        0
        • R Robert Winke

          @Christian-Ehrlicher it's a unique pointer but I have also tried it with a raw pointer

          Christian EhrlicherC Offline
          Christian EhrlicherC Offline
          Christian Ehrlicher
          Lifetime Qt Champion
          wrote on last edited by
          #4

          @Robert-Winke Your code can't work the way you posted it because release() destroys your QFile object.

          Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
          Visit the Qt Academy at https://academy.qt.io/catalog

          R 1 Reply Last reply
          0
          • Christian EhrlicherC Christian Ehrlicher

            @Robert-Winke Your code can't work the way you posted it because release() destroys your QFile object.

            R Offline
            R Offline
            Robert Winke
            wrote on last edited by
            #5

            @Christian-Ehrlicher According to this:
            http://www.cplusplus.com/reference/memory/unique_ptr/release/
            it just releases the responsibility to delete the object but doesn't destroy it.
            But anyway I just tried it again using a raw pointer (by calling new) and also tried it as a stack allocated member variable (passing the address of it to post() and setParent() calls. And it has the same result.

            1 Reply Last reply
            0
            • Christian EhrlicherC Offline
              Christian EhrlicherC Offline
              Christian Ehrlicher
              Lifetime Qt Champion
              wrote on last edited by
              #6

              I misread it as reset(), sorry.
              No need for a member at all.

              QFle *f = new QFile(...);
              if (!f->open(QIODevice::ReadOnly))
                return false;
              _pReply = _netMan.post(postFileRequest, f);
              f->setParent(_pReply); 
              

              Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
              Visit the Qt Academy at https://academy.qt.io/catalog

              R 1 Reply Last reply
              1
              • Christian EhrlicherC Christian Ehrlicher

                I misread it as reset(), sorry.
                No need for a member at all.

                QFle *f = new QFile(...);
                if (!f->open(QIODevice::ReadOnly))
                  return false;
                _pReply = _netMan.post(postFileRequest, f);
                f->setParent(_pReply); 
                
                R Offline
                R Offline
                Robert Winke
                wrote on last edited by
                #7

                @Christian-Ehrlicher said in QNetworkAccessManager doesn't post request in case of large file:

                QFle *f

                I tried this as well but it doesn't make a difference. Still it works with a small file but not with the 6 Mb one

                JonBJ KroMignonK 2 Replies Last reply
                0
                • R Robert Winke

                  @Christian-Ehrlicher said in QNetworkAccessManager doesn't post request in case of large file:

                  QFle *f

                  I tried this as well but it doesn't make a difference. Still it works with a small file but not with the 6 Mb one

                  JonBJ Offline
                  JonBJ Offline
                  JonB
                  wrote on last edited by
                  #8

                  @Robert-Winke
                  Totally throw away observation, no idea if it has any validity: is the scope of your QNetworkRequest relevant, does it have to stay in scope while the upload is in progress?

                  R 1 Reply Last reply
                  0
                  • R Robert Winke

                    @Christian-Ehrlicher said in QNetworkAccessManager doesn't post request in case of large file:

                    QFle *f

                    I tried this as well but it doesn't make a difference. Still it works with a small file but not with the 6 Mb one

                    KroMignonK Offline
                    KroMignonK Offline
                    KroMignon
                    wrote on last edited by
                    #9

                    @Robert-Winke Are you sure the QFile instance is create in same thread as _pReply?
                    You could change the code as follow:

                        auto currentFile = _file.release();
                        _pReply = _netMan.post(postFileRequest, currentFile);
                    
                        ...
                    
                        // capture QFile pointer in lambda and delete it there
                        QObject::connect(_pReply, &QNetworkReply::finished,
                            [this, currentFile ](){ // also catch FP here
                                auto error = this->_pReply->error();
                                currentFile->delateLater();
                                ...
                            }); 
                    

                    It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth. (Sherlock Holmes)

                    R 1 Reply Last reply
                    0
                    • KroMignonK KroMignon

                      @Robert-Winke Are you sure the QFile instance is create in same thread as _pReply?
                      You could change the code as follow:

                          auto currentFile = _file.release();
                          _pReply = _netMan.post(postFileRequest, currentFile);
                      
                          ...
                      
                          // capture QFile pointer in lambda and delete it there
                          QObject::connect(_pReply, &QNetworkReply::finished,
                              [this, currentFile ](){ // also catch FP here
                                  auto error = this->_pReply->error();
                                  currentFile->delateLater();
                                  ...
                              }); 
                      
                      R Offline
                      R Offline
                      Robert Winke
                      wrote on last edited by
                      #10

                      @KroMignon said in QNetworkAccessManager doesn't post request in case of large file:

                      e threa

                      yes, they are in the same thread also it already opens the file just before the post request so it exists.
                      I made your recommended change, it does seem more elegant but my problem persists:(

                      KroMignonK 1 Reply Last reply
                      0
                      • B Offline
                        B Offline
                        Bonnie
                        wrote on last edited by
                        #11

                        Have you tried to upload using other tools?
                        Maybe your server has size limitation for uploads.

                        R 1 Reply Last reply
                        0
                        • R Robert Winke

                          @KroMignon said in QNetworkAccessManager doesn't post request in case of large file:

                          e threa

                          yes, they are in the same thread also it already opens the file just before the post request so it exists.
                          I made your recommended change, it does seem more elegant but my problem persists:(

                          KroMignonK Offline
                          KroMignonK Offline
                          KroMignon
                          wrote on last edited by KroMignon
                          #12

                          @Robert-Winke I read again you first post, you write error 2 which is QNetworkReply::RemoteHostClosedError.

                          This means that the server closed the socket. So perhaps the problem is not on this software but on the server side.
                          Are there logs you can check or do you have tried to look tcp traffic with a network sniffer like WireShark?

                          It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth. (Sherlock Holmes)

                          R 1 Reply Last reply
                          0
                          • KroMignonK KroMignon

                            @Robert-Winke I read again you first post, you write error 2 which is QNetworkReply::RemoteHostClosedError.

                            This means that the server closed the socket. So perhaps the problem is not on this software but on the server side.
                            Are there logs you can check or do you have tried to look tcp traffic with a network sniffer like WireShark?

                            R Offline
                            R Offline
                            Robert Winke
                            wrote on last edited by
                            #13

                            @KroMignon That is a good guess but I forgot to mention that I have a python script which makes this post request with the large file without a problem and the server receives the file so it can't be a server side problem..
                            I am using wireshark to monitor the traffic and in case of large file I don't see the post request (no packets leave my computer) but with the small file I see it and it works

                            KroMignonK 1 Reply Last reply
                            0
                            • R Robert Winke

                              @KroMignon That is a good guess but I forgot to mention that I have a python script which makes this post request with the large file without a problem and the server receives the file so it can't be a server side problem..
                              I am using wireshark to monitor the traffic and in case of large file I don't see the post request (no packets leave my computer) but with the small file I see it and it works

                              KroMignonK Offline
                              KroMignonK Offline
                              KroMignon
                              wrote on last edited by
                              #14

                              @Robert-Winke It look like you hurting same issue as this one ==> https://www.kdab.com/there-and-back-again/

                              It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth. (Sherlock Holmes)

                              R 1 Reply Last reply
                              0
                              • B Bonnie

                                Have you tried to upload using other tools?
                                Maybe your server has size limitation for uploads.

                                R Offline
                                R Offline
                                Robert Winke
                                wrote on last edited by
                                #15

                                @Bonnie Yes I have a working python script so it's not a server side problem but it's a good guess. I edited the original post

                                1 Reply Last reply
                                0
                                • JonBJ JonB

                                  @Robert-Winke
                                  Totally throw away observation, no idea if it has any validity: is the scope of your QNetworkRequest relevant, does it have to stay in scope while the upload is in progress?

                                  R Offline
                                  R Offline
                                  Robert Winke
                                  wrote on last edited by
                                  #16

                                  @JonB I think it should be fine as a local variable but just in case I made it a member var but it didn't help:(

                                  1 Reply Last reply
                                  1
                                  • SGaistS Offline
                                    SGaistS Offline
                                    SGaist
                                    Lifetime Qt Champion
                                    wrote on last edited by
                                    #17

                                    Hi,

                                    Would it be possible to provide a minimal example application and server that shows that issue ?

                                    Interested in AI ? www.idiap.ch
                                    Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                                    R 2 Replies Last reply
                                    1
                                    • KroMignonK KroMignon

                                      @Robert-Winke It look like you hurting same issue as this one ==> https://www.kdab.com/there-and-back-again/

                                      R Offline
                                      R Offline
                                      Robert Winke
                                      wrote on last edited by
                                      #18

                                      @KroMignon I tried with the patch for that issue but it didn't help. That description is mostly just about latency issue. :(

                                      JonBJ 1 Reply Last reply
                                      0
                                      • R Robert Winke

                                        @KroMignon I tried with the patch for that issue but it didn't help. That description is mostly just about latency issue. :(

                                        JonBJ Offline
                                        JonBJ Offline
                                        JonB
                                        wrote on last edited by
                                        #19

                                        @Robert-Winke
                                        If you hope for an answer here, I suggest you have a go at @SGaist's request, we need some minimal sample app to reproduce.

                                        1 Reply Last reply
                                        1
                                        • SGaistS SGaist

                                          Hi,

                                          Would it be possible to provide a minimal example application and server that shows that issue ?

                                          R Offline
                                          R Offline
                                          Robert Winke
                                          wrote on last edited by
                                          #20

                                          @SGaist yes, I created a sandbox project for this with a working web server in python. There is a readme with instructions how to build and test it. I tried to attach the file here but I dont have enough privileges so here is a google drive link:

                                          https://drive.google.com/drive/folders/1lxPun_p_crKlui5KU8Q1qKBp0ue7vvok?usp=sharing

                                          1 Reply Last reply
                                          0

                                          • Login

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Users
                                          • Groups
                                          • Search
                                          • Get Qt Extensions
                                          • Unsolved