Slow POST upload using QtNetworkAccessManager

GregB

I'm writing a Qt5 program to upload files to a server using HTTP and POST. It works fine to send the files, but the speed is really slow. When I upload files using SCP from my development machine to the server, I get ~200Mbps (measured from Windows task manager). When I use an HTML form on the server's website and POST to the same PHP page that my Qt program is posting to, I get ~140Mbps. But when I upload from my Qt application, I get ~10Mbps. The files I'm trying to upload are around 400MB.

I understand there is a difference between SCP and HTTP, but in theory POSTing a file through the browser and through Qt to the same PHP page should be same speed, not 14x slower.

Below is the code that does the uploading

QUrl url(connServer + "/api.php");
QNetworkRequest request(url);

QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);

QHttpPart loginPart;
/* username */
loginPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"u\""));
loginPart.setBody(connUsername.toLatin1());
multiPart->append(loginPart);
/* password */
loginPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"p\""));
loginPart.setBody(connPassword.toLatin1());
multiPart->append(loginPart);

QStringList md5stringlist;
/* loop through the list of files */
ui->progUpload->setRange(0,100);
for (int i=0;i<list.size();i++) {
    qDebug("UploadFileList [%d] [%s]", i, list[i].toStdString().c_str());
    QFile *file = new QFile(list[i]);
    QHttpPart filePart;
    filePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"files[]\"; filename=\""+ file->fileName() + "\""));
    file->open(QIODevice::ReadOnly);
    filePart.setBodyDevice(file);
    file->setParent(multiPart); // we cannot delete the file now, so delete it with the multiPart
    multiPart->append(filePart);
    /* create the MD5 list [file|md5,file2|md5,etc] */
    QFileInfo fileInfo(file->fileName());
    QString filename(fileInfo.fileName());
    md5stringlist << md5list[i];
}

/* check if there was a previous upload still going on */
while (isUploading) {
    ui->lblStatus->setText("Waiting for previous upload to complete...");
    QTest::qWait(1000);
}

/* do the POST and setup the event handlers for it */
QNetworkReply* reply = networkManager->post(request, multiPart);
multiPart->setParent(reply); // delete the multiPart with the reply
numNetConn++;
isUploading = true;
connect(reply, SIGNAL(finished()), this, SLOT(onGetReplyUpload()));
connect(reply, SIGNAL(error(QNetworkReply::NetworkError)), this, SLOT(onNetworkError(QNetworkReply::NetworkError)));
connect(reply, SIGNAL(uploadProgress(qint64, qint64)), SLOT(progressChanged(qint64, qint64)));

What is going on? Am I missing a setting in QtNetworkAccessManager?

raven-worx

@GregB
are you testing a release build and without a debugger attached?

GregB

Yes. Release build on Windows 10. No debugging.

GregB

@raven-worx Yes. Release build on Windows 10. No debugging.

ambershark

@GregB In a way you are comparing 2 different things. I've never seen an HTTP POST upload be anywhere near as fast as my scps.

What I would do is check your apache (or whatever webserver you use) and make sure there isn't a limitation on upload speeds. That is usually a setting to help throttle connections.

Secondly if you really want to narrow things down I would compare a Qt POST upload with a non-Qt POST upload and see if there is a difference. Normally when you see performance stuff like this in files and network traffic it has to do with the buffers being used. It is possible Qt's network buffer is too small for those files.

Or you can go the other route and try an scp upload to your server using Qt's Networking and see if it reaches the same speeds. It should. I wouldn't necessarily do this though as you are not concerned with scp speed but with POST speed.

To limit variables in your testing I would have 2 functions in your app, 1 that uses Qt's HTTP stuff, and one that uses the network directly. This should let you figure out if it's Qt or not. Then you can look at why it's happening with the Qt Http stuff and file a bug if necessary or adjust your code if it's a user error.

Wish I could give more insight but I haven't used Qt's HTTP stuff enough to notice any glaring issues with your code.

GregB

@ambershark Thanks for the suggestions!

I have tried a Qt POST and non-Qt POST to the same PHP page on the same server. Qt is slow, POST through the browser is fast. I can't use scp for this application though because the only port open on the production server is 80, and only httpd is running on it.

The Qt network buffer sounds like a possibility, since the files are so large. How can I change the buffer size?

ambershark

@GregB The part that handles the buffering would be QNetworkReply* reply = networkManager->post(request, multiPart);.

I'd look into your networkManager object first. If that's Qt and assuming it is, check the docs on that and see if there is a way to change it.

If not what I usually do when I run into things like this is actually go look at Qt's code and see if they are doing something stupid like a small buffer or worse yet byte by byte. :) If they are you can of course modify it and rebuild Qt or submit a bug for a larger buffer or buffer size control.

raven-worx

@GregB
just for testing purposes: can you try to upload the file by using a QByteArray (loading the whole file into memory before uploading) instead of the QIODevice-Multipart stuff?

GregB

@raven-worx How do I POST using the QByteArray instead?

This is what I have now

QFile *file = new QFile(list[i]);
QHttpPart filePart;
filePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"files[]\"; filename=\""+ file->fileName() + "\""));
file->open(QIODevice::ReadOnly);
filePart.setBodyDevice(file);
file->setParent(multiPart); // we cannot delete the file now, so delete it with the multiPart
multiPart->append(filePart);

Would this be correct?

QFile *file = new QFile(list[i]);
QHttpPart filePart;
filePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"files[]\"; filename=\""+ file->fileName() + "\""));
filePart.setBody(file->readAll());
multiPart->append(filePart);