qt file uploading, Connection closed error

4j1th

Hi,

I need the application to upload images to our server, it may have 2000-5000 images and each file is about 20-30kb, the application upload one file at a time.
When upload around 30-50 files qt shows an error "Connection closed", I don't know what causing this, my hosting is in Godaddy and I contacted and they there is no problem in the server, my hosting account is a shared server.
If upload again it shows the same error but if uploaded after 20 or 30, I can upload another 10-20 files but it's not a good workaround but now I am using this technique.

void myServer::uploadFiles()
{
    //qDebug() << "start uploading files";

    sqlQuery.exec("SELECT data_contentType, data_createdDate, data_eventId, data_name, data_type, "
                  "data_teamMemberId, dataId FROM data WHERE uploaded  = '0' AND converted = '1' LIMIT 1 ");
    if (sqlQuery.next()) {
        qDebug() << "uploading : "+sqlQuery.value(3).toString();

        QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);

        QHttpPart data_contentType;
        data_contentType.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"data_contentType\""));
        data_contentType.setBody(sqlQuery.value(0).toString().toLatin1());

        QHttpPart data_createdDate;
        data_createdDate.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"data_createdDate\""));
        data_createdDate.setBody(sqlQuery.value(1).toString().toLatin1());

        QHttpPart data_eventId;
        data_eventId.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"data_eventId\""));
        data_eventId.setBody(sqlQuery.value(2).toString().toLatin1());

        QHttpPart data_name;
        data_name.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"data_name\""));
        data_name.setBody(sqlQuery.value(3).toString().toLatin1());

        QHttpPart data_type;
        data_type.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"data_type\""));
        data_type.setBody(sqlQuery.value(4).toString().toLatin1());

        QHttpPart data_teamMemberId;
        data_teamMemberId.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"data_teamMemberId\""));
        data_teamMemberId.setBody(sqlQuery.value(5).toString().toLatin1());

        QHttpPart dataId;
        dataId.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"dataId\""));
        dataId.setBody(sqlQuery.value(6).toString().toLatin1());

        QHttpPart imagePart;
        imagePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("image/jpeg"));
        imagePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name=\"image\"; filename=\"1.jpg\""));
        QFile *file = new QFile(settings.value("app/workingDir").toString()+"w_500/"+sqlQuery.value(3).toString());
        file->open(QIODevice::ReadOnly);
        imagePart.setBodyDevice(file);
        file->setParent(multiPart); // we cannot delete the file now, so delete it with the multiPart

        multiPart->append(data_contentType);
        multiPart->append(data_createdDate);
        multiPart->append(data_eventId);
        multiPart->append(data_name);
        multiPart->append(data_type);
        multiPart->append(data_teamMemberId);
        multiPart->append(dataId);
        multiPart->append(imagePart);

        QUrl url(settings.value("app/serverUrl").toString()+"update/upload_files");
        QNetworkRequest request(url);

        //QNetworkAccessManager manager;
        QNetworkAccessManager * manager = new QNetworkAccessManager();
        QNetworkReply *reply = manager->post(request, multiPart);
        multiPart->setParent(reply); // delete the multiPart with the reply

        // here connect signals etc.
        connect(manager, SIGNAL(finished(QNetworkReply*)), SLOT(fileUploadFinished(QNetworkReply*)));
    }
}

void myServer::fileUploadFinished(QNetworkReply *replay)
{
    // Checking replay
    if (replay->isReadable()) {
        // Checking errors
        if(replay->error() == QNetworkReply::NoError) {
            QByteArray strreplay = replay->readAll();

            // update ststus check
            if(strreplay != "file uploading failed") {

                qDebug() << "upload sucessfull "+strreplay;

                sqlQuery.exec("SELECT data_name FROM data WHERE dataId = '"+strreplay+"'");
                if (sqlQuery.next()) {

                    // remove converted image
                    QFile imgfile(settings.value("app/workingDir").toString()+"w_500/"+sqlQuery.value(0).toString());
                    imgfile.remove();

                    // update database
                    sqlQuery.exec("UPDATE data SET uploaded  = '1' WHERE dataId = '"+strreplay+"' ");
                }
            } else {
                qDebug() << strreplay;
            }
        } else {
            qDebug() << replay->errorString();
            QTimer::singleShot(20000, this, SLOT(waitupload()));
            return;
        }
    }
    replay->close();

    this->uploadFiles();
}

Thanks

Stoyan

@4j1th
I'm not sure if this is the reason, but you created a lot of objects QNetworkAccessManager and QNetworkReply and never delete them. According to documentation:
"One QNetworkAccessManager should be enough for the whole Qt application."
"Currently, for the HTTP protocol on desktop platforms, 6 requests are executed in parallel for one host/port combination."
May be it is better to make QNetworkAccessManager member of the class myServer and to not create new object in uploadFiles.

QNetworkAccessManager *manager;  // in header file

manager = new QNetworkAccessManager();  // In the constructor of myServer

myServer::uploadFiles()
...
QNetworkReply *reply = manager->post(request, multiPart);

And to delete QNetworkReply objects:

fileUploadFinished(QNetworkReply *replay)
...
replay->close();
replay->deleteLater();

And what is doing the slot waitupload()?

4j1th

@Stoyan Thanks for the replay

I tried you said but it save same file copy multiple times in server and don't return the send post data in order

4j1th

@Stoyan

waitupload() - used as a workaround of this error, the execution waits 20 seconds and continue the upload and it works as i said in my question

4j1th

@Stoyan I think this post solves the new object issue

https://stackoverflow.com/questions/23267925/when-to-delete-qnetworkaccessmanager-pointer

// Delete object whose signal triggered this slot.
    QObject *manager = sender();
    manager->deleteLater();

But this doesn't solve the connection error issue

Stoyan

@4j1th
Did you add

replay->deleteLater();

What is the result of replay->errorString(); ?
The code you post should work if you create instance of QNetworkAccessManager every time.
Did you left in code this row:

QNetworkAccessManager * manager = new QNetworkAccessManager();

Did you call function uploadFiles() from multiple places?
I suspect that you create multiple connection to server and there is a limitation for such activity.

Stoyan

@4j1th said in qt file uploading, Connection closed error:

@Stoyan Thanks for the replay

I tried you said but it save same file copy multiple times in server and don't return the send post data in order

I think I understand why this happens. Try to move this row

connect(manager, SIGNAL(finished(QNetworkReply*)), SLOT(fileUploadFinished(QNetworkReply*)));

in the constructor, right after this:

manager = new QNetworkAccessManager();

Else it creates multiple connections for one object.

4j1th

@Stoyan now it's working but still generate the "Connection closed" error from 'replay->errorString();'

Stoyan

@4j1th
Can you put in the beginning of fileUploadFinished, before checking for errors, this code:

QByteArray strreplay = replay->readAll();
qDebug() << "Reply: " << strreplay;
qDebug() << "Reply error code: " << replay->error();

Maybe there is some informative response to some of the requests before terminating connection.
This is the list with all error codes: QNetworkReply::NetworkError

4j1th

@Stoyan thank you for that debugging idea

"uploading : 2017_06_18_00_00_00_26/raw/8/images/mUarFdVu5ZZmiOqFjLoTz4lXn.JPG"
Reply:  "27"
Reply error code:  QNetworkReply::NetworkError(NoError)
"upload sucessfull 27"
"uploading : 2017_06_18_00_00_00_26/raw/8/images/sj0DMlogCTmdoUZEtBk8pph0R.JPG"
Reply:  "28"
Reply error code:  QNetworkReply::NetworkError(NoError)
"upload sucessfull 28"
"uploading : 2017_06_18_00_00_00_26/raw/8/images/0nMaAxIhlJtNxOPHystIFv2EW.JPG"
Reply:  "29"
Reply error code:  QNetworkReply::NetworkError(NoError)
"upload sucessfull 29"
"uploading : 2017_06_18_00_00_00_26/raw/8/images/0t9VhPJKZjIMppyzZBknOpb4Z.JPG"
Reply:  "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>408 Request Timeout</title>\n</head><body>\n<h1>Request Timeout</h1>\n<p>Server timeout waiting for the HTTP request from the client.</p>\n<p>Additionally, a 408 Request Timeout\nerror was encountered while trying to use an ErrorDocument to handle the request.</p>\n</body></html>\n"
Reply error code:  QNetworkReply::NetworkError(UnknownContentError)
"Error transferring http://client.mydomain.com/index.php/update/upload_files - server replied: Request Timeout"

4j1th

@Stoyan another error now I get is

Reply error code: QNetworkReply::NetworkError(RemoteHostClosedError)

Stoyan

@4j1th
If error "RemoteHostClosedError" follow the previous error "UnknownContentError", probably the reason is the first error.
You may run the program some more times to collect more data and contact again your service provider with this information.
It is possible to exists some limitation on your account. For example: number of transferred files for a period. Some servers after too many requests for a short time, ask for some type additional identification (captcha for example).

And one more thing (not directly connected to the main problem) - you should close and delete the instance of "replay" even when an error occurred.

4j1th

@Stoyan thanks for the help, I also think that this is a server limitation issue I am trying to resolve the issue with Godaddy, mean time I think the workaround is useful.