Using QHttpMultiPart to send big files via HTTP freezes my application

Creaperdown

Hey, my client application needs to send files of ~1-100MB to my rest api which expects a FormFile. I have looked around for a bit and found QHttpMultiPart, which seemed to be a reasonable solution to my problem.

I have tried a small example to see if it is working correctly. It seems to work just fine for small files (.txt files with some words of content), but if the file is getting a bit bigger (100KB pdf file) my Qt application just freezes.

This is the code that I have been using:

// Open file
    QFile* file = new QFile("/home/creapermann/Downloads/api-design.pdf");
    if(!file->open(QIODevice::ReadOnly | QIODevice::Text))
    {
        qDebug() << "Could not open test file!";
        return;
    }

    QHttpMultiPart* multiPart =
        new QHttpMultiPart(QHttpMultiPart::FormDataType);


    QHttpPart filePart;
    filePart.setHeader(QNetworkRequest::ContentTypeHeader,
                       QVariant("multipart/form-data"));
    filePart.setHeader(QNetworkRequest::ContentDispositionHeader,
                       QVariant("form-data; name=\"file\"; filename=\"" +
                                file->fileName() + "\""));
    filePart.setBodyDevice(file);
    file->setParent(multiPart);
    multiPart->append(filePart);


    QUrl url(serverDomain);
    QNetworkRequest testRequest { url };

    // -- Some Server Specific Headers --


// -- Some SSL configuration code --


    auto reply = m_networkAccessManager.post(testRequest, multiPart);
    m_testReply.reset(reply);

    linkRequestToErrorHandling(m_testReply.get(), 201);

Does anyone have an idea why it might not work correctly?
Thanks in advance

Creaperdown

I have found the problem...

I have called: file->open(QIODevice::ReadOnly | QIODevice::Text) while the file I am reading is not Text, opening the file simply via: file->open(QIODevice::ReadOnly) fixed it.

Kent-Dorfman

@Creaperdown
Cannot answer authoritatively but I'd suspect that multipart/form-data is not the correct way to adhoc binary files. Ever hear of out-of-band transmission? declare your communications channel using the rest api, but send it by another means: ie ssh, straight binary blob, etc.

Creaperdown

@Kent-Dorfman Is there any support from Qt for this, or would I need to write it from scratch?

Kent-Dorfman

@Creaperdown
I dunno. Research it!

JonB

@Creaperdown
Certainly there is nothing "magical" in Qt to do it.

Not my area, but I would start by Googling for, say, can you send big files via http. You state you have a REST API, one hit is https://stackoverflow.com/questions/33889410/proper-way-to-implement-restful-large-file-upload, does this give useful suggestions?

Creaperdown

@JonB it seems to be a reasonable approach to use QHttpMultipart to send big files via http. I have spoken to multiple people which said that they do this in their projects and it works just fine.

The problem of my application freezing when using QHttpMultipart still remains though. I have created a StackOverflow question where I elaborated my problem a bit more: https://stackoverflow.com/questions/75910403/sending-files-via-qhttpmultipart-freezes if anyone needs more information about my problem.

ChrisW67

@Creaperdown said in Using QHttpMultiPart to send big files via HTTP freezes my application:

QHttpMultiPart* multiPart =
       new QHttpMultiPart(QHttpMultiPart::FormDataType);


   QHttpPart filePart;
   filePart.setHeader(QNetworkRequest::ContentTypeHeader,
                      QVariant("multipart/form-data"));

This is incorrect. The entire multipart construct is of MIME type "multipart/form-data" (set by the type passed to QHttpMultiPart constructor). Each individual part (QHttpPart) has its own MIME type that may be specific or generic binary data e.g. "image/jpeg" or "application/octet-stream". You do not set a type for your binary data below, which leads to the assumption of "text/plain".

   filePart.setHeader(QNetworkRequest::ContentDispositionHeader,
                       QVariant("form-data; name=\"file\"; filename=\"" +
                                file->fileName() + "\""));

Read the example in the QHttpMultipart docs.
Sending a malformed request to a server will usually result in a 400 HTTP error code, but may also cause the server issues (500 series code) or cause a firewall/IPS to simply drop the request. Sending too large a payload may also be rejected.

None of this should hang your program.

Creaperdown

@ChrisW67 So if I understand it correctly, I am assigning the wrong MIME type to my filePart?

Does this mean that changing it to:

 QHttpPart filePart;
 filePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("application/octet-stream"));

Should fix it?

ChrisW67

@Creaperdown Probably, but I do not know what your server is expecting.

Creaperdown

@ChrisW67 I tried it but it does not change anything. My server is expecting a multi-part request in the body of the HTTP request. It manages to find it and read it for small files with just one chunk, but when there are multiple chunks, the server gets stuck waiting for the 2nd, since it never seems to get sent

ChrisW67

Does you code return to the Qt event loop?
Does the QFile object survive long enough for the transfer to occur? (The pointer to it probably goes out of scope but the object can persist).

Creaperdown

@ChrisW67 In the code I sent, the file's parent is the QHttpMultipart object and this multipart object is created on the heap and is never deleted, it is basically leaking but that is fine since its just a test and I wanted to make sure that there is no lifetime issue.

Creaperdown

@ChrisW67 Also, how would I check if the code returns to the Qt event loop?

JonB

@Creaperdown
For that one what path does your code follow after you have posted the request? Have you allowed it it to e.g. return into the QApplication::exec() started from your main() function?

Creaperdown

@JonB I am not sure what you mean with "Have you allowed it it to e.g. return into the QApplication::exec()".
How would I allow / forbid it?

Afaik. I have not forbidden my application to return back to the event loop.

JonB

@Creaperdown
One way people (accidentally) prevent it is by running any kind of while (true) ... loop to do something. Or they write sleep(1000000).

Where is the code you show called from? Is it e.g. called from a slot in a UI? Then so long as you allow the slot to return that would be OK. Maybe you don't have a UI, maybe your application is command line? What (if anything) does your code do after this code has executed? We don't know.

Creaperdown

@JonB The function is called by qml code which looks like this:

FileDialog
{
    ...
    onAccepted: BookController.addBook(path);
}

This calls the c++ addBook method which looks like this:

int BookController::addBook(const QString& path)
{
    auto result = m_bookService->addBook(path);
    return static_cast<int>(result);
}

Which in turn calls the next method:

BookOperationStatus BookService::addBook(const QString& filePath)
{
   // Do some stuff....

    const Book& bookToStore = m_books.at(m_books.size() - 1);
    m_bookStorageManager->addBook(bookToStore);  // <--- Store the book

    return BookOperationStatus::Success;
}

Which calls

void BookStorageManager::addBook(const Book& bookToAdd)
{
    // Store book to local file

    m_bookStorageGateway->createBook(m_authenticationToken, bookToAdd);  // <--- Send book to server
}

Which calls:

void BookStorageGateway::createBook(const QString& authToken, const Book& book)
{
    auto jsonDoc = QJsonDocument::fromJson(book.toJson());
    auto jsonBook = jsonDoc.object();

    convertJsonBookToApiFormat(jsonBook);

    m_bookStorageAccess->createBook(authToken, jsonBook);
}

And which then finally calls the code that I have shared before:

void BookStorageAccess::createBook(const QString& authToken,
                                   const QJsonObject& jsonBook)
{
   // Create multi-part
    m_multiPart.reset(new QHttpMultiPart(QHttpMultiPart::FormDataType));


    // Open file
    QFile* file = new QFile(QUrl(jsonBook["filePath"].toString()).path());
    if(!file->open(QIODevice::ReadOnly | QIODevice::Text))
    {
        qDebug() << "Could not open test file!";
        return;
    }


    // Create file part
    QHttpPart filePart;
    filePart.setHeader(QNetworkRequest::ContentTypeHeader,
                       QVariant("application/pdf"));
    filePart.setHeader(QNetworkRequest::ContentDispositionHeader,
                       QVariant("form-data; name=\"file\"; filename=\"" +
                                file->fileName() + "\""));

    filePart.setBodyDevice(file);
    file->setParent(m_multiPart.get());


    // Add file part
    m_multiPart->append(filePart);


    // Setup request
    QUrl url(data::baseUrl + "/api/book/data");
    QNetworkRequest testRequest { url };
    testRequest.setRawHeader("X-Version", "1.0");
    testRequest.setRawHeader(QByteArray("Authorization"),
                             "Bearer " + authToken.toUtf8());
    testRequest.setHeader(QNetworkRequest::ContentDispositionHeader,
                          QVariant("form-data; name=\"file\"; filename=\"" +
                                   file->fileName() + "\""));


    // Configure SSL
    QSslConfiguration sslConfiguration = testRequest.sslConfiguration();
    sslConfiguration.setProtocol(QSsl::AnyProtocol);
    sslConfiguration.setPeerVerifyMode(QSslSocket::QueryPeer);
    testRequest.setSslConfiguration(sslConfiguration);


    // Send request
    auto reply = m_networkAccessManager.post(testRequest, m_multiPart.get());
    m_testReply.reset(reply);


    connect(m_testReply.get(), &QNetworkReply::uploadProgress,
            [](qint64 bytesSent, qint64 bytesTotal)
            {
                qDebug() << QString::number(bytesSent) + " of " +
                                QString::number(bytesTotal) + " uploaded";
            });

    connect(m_testReply.get(), &QNetworkReply::finished,
            []()
            {
                qDebug() << "Upload finished!";
            });
}

Note that I have changed the last method a bit compared to the code that I have provided before. I tried some things, but none of them fixed my problem.

JonB

@Creaperdown
Firstly this is the first time you have mentioned you are using QML. It's probably fine, but I know nothing about QML.

This may all be a rabbit hole, but I don't know what your m_multiPart & m_testReply are/how they work. From my knowledge of C++ I don't know how you get

m_multiPart.get()
// together with
m_multiPart->append(filePart)

to work (compile). [Maybe it works if get() is a static method of YourDerivedHttpMultiPart so equivalent to YourDerivedHttpMultiPart::get(), I don't know.] The important thing is that necessary objects have a lifetime across the file transfer. If I wanted to be sure I might put connect(object, &QObject::destroyed, this, [object]() { qDebug() << object << " destroyed"; } ) on those objects derived from QObject you create/use to make sure I knew when they get destroyed. If they are somehow "short-lived" maybe it's possible they send the first "block" of the transfer but no more.

Creaperdown

@JonB m_multiPart is a std::unique_ptr<QHttpMultiPart> in the header file of my class. I'll now try connecting a slot to the destroyed signal of QObject to know when they are destroyed

Creaperdown

@JonB I have now added:

// Create multi-part
    m_multiPart.reset(new QHttpMultiPart(QHttpMultiPart::FormDataType));
    connect(m_multiPart.get(), &QObject::destroyed,
            []()
            {
                qDebug() << "multipart destroyed!";
            });


    // Open file
    QFile* file = new QFile(QUrl(jsonBook["filePath"].toString()).path());
    if(!file->open(QIODevice::ReadOnly | QIODevice::Text))
    {
        qDebug() << "Could not open test file!";
        return;
    }
    connect(file, &QObject::destroyed,
            []()
            {
                qDebug() << "file destroyed!";
            });

But nothing is ever printed, thus the objects should never get destroyed