QNetworkAccessManager doesn't post request in case of large file

[Robert Winke]

Hello everyone,
I'm trying to make a post request for a larger (~6Mb) text file with QNetworkAccessManager. The reply is connected to a lambda which should print out the progress however after very little (0.1%) progress I get Connection closed error (error code: 2). I'm monitoring the network traffic with wireshark and I don't even see the post request there

HOWEVER if I use a small txt file then everything works fine (and I can see the request from the monitoring tool).

So the only difference between the working and non-working scenario is the file size.

Here is the relevant code snippet:

void Gen2FirmwareUpdater::_UploadFWImage()
{
    QString url = _restUrl.arg(_ipAddress).arg("upload");

    std::cout << "Uploading FW file to: " + url.toStdString() + "\n";
    QNetworkRequest postFileRequest(url);

    //postFileReqauest.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");
    postFileRequest.setHeader(QNetworkRequest::ContentTypeHeader, "text/plain");
    postFileRequest.setHeader(QNetworkRequest::ContentLengthHeader, _file->size());

    if (!_file->open(QIODevice::ReadOnly)) {
        std::cout << "Unable to open file\n ";
    }

    _pReply = _netMan.post(postFileRequest, _file.get());
    _file.release()->setParent(_pReply);   // now the reply will take care of
                                           // deleting the file handle

    auto onProgUpdate =
    [this](auto sent, auto total){
        if (total != 0) {
            auto prog = sent*100/(2*static_cast<double>(total));
            std::cout << "prog: " << prog << "\n";
        }
    };

    // Queue up the progress update so it doesn't intervene
    // with the underlying process
    QObject::connect(_pReply, &QNetworkReply::uploadProgress,
                     _pReply, onProgUpdate, Qt::QueuedConnection);

    QObject::connect(_pReply, &QNetworkReply::finished,
        [this](){
            auto error = this->_pReply->error();
            if (error) {
                auto msg =
                "Error Code: "   + std::to_string(static_cast<int>(error)) + "\n"
                "\nError String: " + this->_pReply->errorString().toStdString();
                std::cout << msg << "\n";
            } else {
                std::cout << "New FW uploaded\n";
            }
            auto r = this->_pReply->readAll();
            if (r.size() > 0) {
                std::cout << "Server response: " + r.toStdString() + "\n\n";
            }
        });

    //QEventLoop loop;
    //QObject::connect(_pReply, &QNetworkReply::finished, &loop, &QEventLoop::quit);
    //loop.exec();
}

I suspected that maybe something gets deleted before the upload completes so I'm using member variables for the network access manager and the reply (which should take ownership of the file).

I have a python script which can do this upload so that eliminates server side problems.

I also tried using QHttpMultiPart (not even sure that that is meant for this) but I had the same results.

Here is a sandbox project with a working web server and readme on how to use it.
https://drive.google.com/drive/folders/1lxPun_p_crKlui5KU8Q1qKBp0ue7vvok?usp=sharing
(using this web server I receive Error Code: 99 Unknown error in case of large file but I can see the post reqest from wireshark, small file version still works perfectly)

I hope there is someone to help me, thanks!

[Christian Ehrlicher]

What is _file? A shared pointer? Why?`And if - you know what release() does?

[Robert Winke]

@Christian-Ehrlicher it's a unique pointer but I have also tried it with a raw pointer

[Christian Ehrlicher]

@Robert-Winke Your code can't work the way you posted it because release() destroys your QFile object.

[Robert Winke]

@Christian-Ehrlicher According to this:
http://www.cplusplus.com/reference/memory/unique_ptr/release/
it just releases the responsibility to delete the object but doesn't destroy it.
But anyway I just tried it again using a raw pointer (by calling new) and also tried it as a stack allocated member variable (passing the address of it to post() and setParent() calls. And it has the same result.

[Christian Ehrlicher]

I misread it as reset(), sorry.
No need for a member at all.

QFle *f = new QFile(...);
if (!f->open(QIODevice::ReadOnly))
  return false;
_pReply = _netMan.post(postFileRequest, f);
f->setParent(_pReply); 

[Robert Winke]

@Christian-Ehrlicher said in QNetworkAccessManager doesn't post request in case of large file:

QFle *f

I tried this as well but it doesn't make a difference. Still it works with a small file but not with the 6 Mb one

[JonB]

@Robert-Winke
Totally throw away observation, no idea if it has any validity: is the scope of your QNetworkRequest relevant, does it have to stay in scope while the upload is in progress?

[KroMignon]

@Robert-Winke Are you sure the QFile instance is create in same thread as _pReply?
You could change the code as follow:

    auto currentFile = _file.release();
    _pReply = _netMan.post(postFileRequest, currentFile);

    ...

    // capture QFile pointer in lambda and delete it there
    QObject::connect(_pReply, &QNetworkReply::finished,
        [this, currentFile ](){ // also catch FP here
            auto error = this->_pReply->error();
            currentFile->delateLater();
            ...
        }); 

[Robert Winke]

@KroMignon said in QNetworkAccessManager doesn't post request in case of large file:

e threa

yes, they are in the same thread also it already opens the file just before the post request so it exists.
I made your recommended change, it does seem more elegant but my problem persists:(

[Bonnie]

Have you tried to upload using other tools?
Maybe your server has size limitation for uploads.

[KroMignon]

@Robert-Winke I read again you first post, you write error 2 which is QNetworkReply::RemoteHostClosedError.

This means that the server closed the socket. So perhaps the problem is not on this software but on the server side.
Are there logs you can check or do you have tried to look tcp traffic with a network sniffer like WireShark?

[Robert Winke]

@KroMignon That is a good guess but I forgot to mention that I have a python script which makes this post request with the large file without a problem and the server receives the file so it can't be a server side problem..
I am using wireshark to monitor the traffic and in case of large file I don't see the post request (no packets leave my computer) but with the small file I see it and it works

[KroMignon]

@Robert-Winke It look like you hurting same issue as this one ==> https://www.kdab.com/there-and-back-again/

[Robert Winke]

@Bonnie Yes I have a working python script so it's not a server side problem but it's a good guess. I edited the original post

[Robert Winke]

@JonB I think it should be fine as a local variable but just in case I made it a member var but it didn't help:(

[SGaist]

Hi,

Would it be possible to provide a minimal example application and server that shows that issue ?

[Robert Winke]

@KroMignon I tried with the patch for that issue but it didn't help. That description is mostly just about latency issue. :(

[JonB]

@Robert-Winke
If you hope for an answer here, I suggest you have a go at @SGaist's request, we need some minimal sample app to reproduce.

[Robert Winke]

@SGaist yes, I created a sandbox project for this with a working web server in python. There is a readme with instructions how to build and test it. I tried to attach the file here but I dont have enough privileges so here is a google drive link:

https://drive.google.com/drive/folders/1lxPun_p_crKlui5KU8Q1qKBp0ue7vvok?usp=sharing

[Robert Winke]

@SGaist ps the error with this web server is not the same. This time I can see the post request from wireshark but the client receives Error code: 99 Unknown error. The progress still doesn't go up to 50 (100%) but it does with the small.txt

[SGaist]

Your sample Python server does not handle post requests properly and kills the connection early.

Once that fixed, big files are properly uploaded.

Now the question is:
auto prog = sent*100/(2*static_cast<double>(total));

What does that 2 do in there ? Mathematically, you can't reach 100% with that equation unless you send twice the number of bytes.

[Robert Winke]

@SGaist That 2 doesn't matter, I'm not expecting it to go to 100 just 50 I just forgot to remove it for this sandbox project.
I'll look into the python server but that's not actually the server I used for testing. The one I used is proven to be working with another client but I cannot attach that.

[Robert Winke]

@SGaist okay by changing the server it does seem to work but now I'm completely lost because the server I used before works too just not with my client...

[KroMignon]

@Robert-Winke said in QNetworkAccessManager doesn't post request in case of large file:

okay by changing the server it does seem to work

I am just curious to know what to you have changed in server code. At first look, I can not see what wrong with it.
Can you please show me the changes?

[Robert Winke]

@KroMignon I uploaded the new server code in the same zip