Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Using QHttpMultiPart to send big files via HTTP freezes my application
Forum Updated to NodeBB v4.3 + New Features

Using QHttpMultiPart to send big files via HTTP freezes my application

Scheduled Pinned Locked Moved Solved General and Desktop
21 Posts 4 Posters 2.2k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • JonBJ JonB

    @Creaperdown
    Certainly there is nothing "magical" in Qt to do it.

    Not my area, but I would start by Googling for, say, can you send big files via http. You state you have a REST API, one hit is https://stackoverflow.com/questions/33889410/proper-way-to-implement-restful-large-file-upload, does this give useful suggestions?

    C Offline
    C Offline
    Creaperdown
    wrote on last edited by
    #6

    @JonB it seems to be a reasonable approach to use QHttpMultipart to send big files via http. I have spoken to multiple people which said that they do this in their projects and it works just fine.

    The problem of my application freezing when using QHttpMultipart still remains though. I have created a StackOverflow question where I elaborated my problem a bit more: https://stackoverflow.com/questions/75910403/sending-files-via-qhttpmultipart-freezes if anyone needs more information about my problem.

    1 Reply Last reply
    0
    • C Creaperdown

      Hey, my client application needs to send files of ~1-100MB to my rest api which expects a FormFile. I have looked around for a bit and found QHttpMultiPart, which seemed to be a reasonable solution to my problem.

      I have tried a small example to see if it is working correctly. It seems to work just fine for small files (.txt files with some words of content), but if the file is getting a bit bigger (100KB pdf file) my Qt application just freezes.

      This is the code that I have been using:

      // Open file
          QFile* file = new QFile("/home/creapermann/Downloads/api-design.pdf");
          if(!file->open(QIODevice::ReadOnly | QIODevice::Text))
          {
              qDebug() << "Could not open test file!";
              return;
          }
      
          QHttpMultiPart* multiPart =
              new QHttpMultiPart(QHttpMultiPart::FormDataType);
      
      
          QHttpPart filePart;
          filePart.setHeader(QNetworkRequest::ContentTypeHeader,
                             QVariant("multipart/form-data"));
          filePart.setHeader(QNetworkRequest::ContentDispositionHeader,
                             QVariant("form-data; name=\"file\"; filename=\"" +
                                      file->fileName() + "\""));
          filePart.setBodyDevice(file);
          file->setParent(multiPart);
          multiPart->append(filePart);
      
      
          QUrl url(serverDomain);
          QNetworkRequest testRequest { url };
      
          // -- Some Server Specific Headers --
      
      
      // -- Some SSL configuration code --
      
      
          auto reply = m_networkAccessManager.post(testRequest, multiPart);
          m_testReply.reset(reply);
      
          linkRequestToErrorHandling(m_testReply.get(), 201);
      

      Does anyone have an idea why it might not work correctly?
      Thanks in advance

      C Offline
      C Offline
      ChrisW67
      wrote on last edited by
      #7

      @Creaperdown said in Using QHttpMultiPart to send big files via HTTP freezes my application:

      QHttpMultiPart* multiPart =
             new QHttpMultiPart(QHttpMultiPart::FormDataType);
      
      
         QHttpPart filePart;
         filePart.setHeader(QNetworkRequest::ContentTypeHeader,
                            QVariant("multipart/form-data"));
      

      This is incorrect. The entire multipart construct is of MIME type "multipart/form-data" (set by the type passed to QHttpMultiPart constructor). Each individual part (QHttpPart) has its own MIME type that may be specific or generic binary data e.g. "image/jpeg" or "application/octet-stream". You do not set a type for your binary data below, which leads to the assumption of "text/plain".

         filePart.setHeader(QNetworkRequest::ContentDispositionHeader,
                             QVariant("form-data; name=\"file\"; filename=\"" +
                                      file->fileName() + "\""));
      

      Read the example in the QHttpMultipart docs.
      Sending a malformed request to a server will usually result in a 400 HTTP error code, but may also cause the server issues (500 series code) or cause a firewall/IPS to simply drop the request. Sending too large a payload may also be rejected.

      None of this should hang your program.

      C 1 Reply Last reply
      1
      • C ChrisW67

        @Creaperdown said in Using QHttpMultiPart to send big files via HTTP freezes my application:

        QHttpMultiPart* multiPart =
               new QHttpMultiPart(QHttpMultiPart::FormDataType);
        
        
           QHttpPart filePart;
           filePart.setHeader(QNetworkRequest::ContentTypeHeader,
                              QVariant("multipart/form-data"));
        

        This is incorrect. The entire multipart construct is of MIME type "multipart/form-data" (set by the type passed to QHttpMultiPart constructor). Each individual part (QHttpPart) has its own MIME type that may be specific or generic binary data e.g. "image/jpeg" or "application/octet-stream". You do not set a type for your binary data below, which leads to the assumption of "text/plain".

           filePart.setHeader(QNetworkRequest::ContentDispositionHeader,
                               QVariant("form-data; name=\"file\"; filename=\"" +
                                        file->fileName() + "\""));
        

        Read the example in the QHttpMultipart docs.
        Sending a malformed request to a server will usually result in a 400 HTTP error code, but may also cause the server issues (500 series code) or cause a firewall/IPS to simply drop the request. Sending too large a payload may also be rejected.

        None of this should hang your program.

        C Offline
        C Offline
        Creaperdown
        wrote on last edited by
        #8

        @ChrisW67 So if I understand it correctly, I am assigning the wrong MIME type to my filePart?

        Does this mean that changing it to:

         QHttpPart filePart;
         filePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("application/octet-stream"));
        

        Should fix it?

        C 1 Reply Last reply
        0
        • C Creaperdown

          @ChrisW67 So if I understand it correctly, I am assigning the wrong MIME type to my filePart?

          Does this mean that changing it to:

           QHttpPart filePart;
           filePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("application/octet-stream"));
          

          Should fix it?

          C Offline
          C Offline
          ChrisW67
          wrote on last edited by
          #9

          @Creaperdown Probably, but I do not know what your server is expecting.

          C 1 Reply Last reply
          0
          • C ChrisW67

            @Creaperdown Probably, but I do not know what your server is expecting.

            C Offline
            C Offline
            Creaperdown
            wrote on last edited by
            #10

            @ChrisW67 I tried it but it does not change anything. My server is expecting a multi-part request in the body of the HTTP request. It manages to find it and read it for small files with just one chunk, but when there are multiple chunks, the server gets stuck waiting for the 2nd, since it never seems to get sent

            C 1 Reply Last reply
            0
            • C Creaperdown

              @ChrisW67 I tried it but it does not change anything. My server is expecting a multi-part request in the body of the HTTP request. It manages to find it and read it for small files with just one chunk, but when there are multiple chunks, the server gets stuck waiting for the 2nd, since it never seems to get sent

              C Offline
              C Offline
              ChrisW67
              wrote on last edited by
              #11

              Does you code return to the Qt event loop?
              Does the QFile object survive long enough for the transfer to occur? (The pointer to it probably goes out of scope but the object can persist).

              C 2 Replies Last reply
              0
              • C ChrisW67

                Does you code return to the Qt event loop?
                Does the QFile object survive long enough for the transfer to occur? (The pointer to it probably goes out of scope but the object can persist).

                C Offline
                C Offline
                Creaperdown
                wrote on last edited by Creaperdown
                #12

                @ChrisW67 In the code I sent, the file's parent is the QHttpMultipart object and this multipart object is created on the heap and is never deleted, it is basically leaking but that is fine since its just a test and I wanted to make sure that there is no lifetime issue.

                1 Reply Last reply
                0
                • C ChrisW67

                  Does you code return to the Qt event loop?
                  Does the QFile object survive long enough for the transfer to occur? (The pointer to it probably goes out of scope but the object can persist).

                  C Offline
                  C Offline
                  Creaperdown
                  wrote on last edited by
                  #13

                  @ChrisW67 Also, how would I check if the code returns to the Qt event loop?

                  JonBJ 1 Reply Last reply
                  0
                  • C Creaperdown

                    @ChrisW67 Also, how would I check if the code returns to the Qt event loop?

                    JonBJ Offline
                    JonBJ Offline
                    JonB
                    wrote on last edited by
                    #14

                    @Creaperdown
                    For that one what path does your code follow after you have posted the request? Have you allowed it it to e.g. return into the QApplication::exec() started from your main() function?

                    C 1 Reply Last reply
                    0
                    • JonBJ JonB

                      @Creaperdown
                      For that one what path does your code follow after you have posted the request? Have you allowed it it to e.g. return into the QApplication::exec() started from your main() function?

                      C Offline
                      C Offline
                      Creaperdown
                      wrote on last edited by
                      #15

                      @JonB I am not sure what you mean with "Have you allowed it it to e.g. return into the QApplication::exec()".
                      How would I allow / forbid it?

                      Afaik. I have not forbidden my application to return back to the event loop.

                      JonBJ 1 Reply Last reply
                      0
                      • C Creaperdown

                        @JonB I am not sure what you mean with "Have you allowed it it to e.g. return into the QApplication::exec()".
                        How would I allow / forbid it?

                        Afaik. I have not forbidden my application to return back to the event loop.

                        JonBJ Offline
                        JonBJ Offline
                        JonB
                        wrote on last edited by
                        #16

                        @Creaperdown
                        One way people (accidentally) prevent it is by running any kind of while (true) ... loop to do something. Or they write sleep(1000000).

                        Where is the code you show called from? Is it e.g. called from a slot in a UI? Then so long as you allow the slot to return that would be OK. Maybe you don't have a UI, maybe your application is command line? What (if anything) does your code do after this code has executed? We don't know.

                        C 1 Reply Last reply
                        0
                        • JonBJ JonB

                          @Creaperdown
                          One way people (accidentally) prevent it is by running any kind of while (true) ... loop to do something. Or they write sleep(1000000).

                          Where is the code you show called from? Is it e.g. called from a slot in a UI? Then so long as you allow the slot to return that would be OK. Maybe you don't have a UI, maybe your application is command line? What (if anything) does your code do after this code has executed? We don't know.

                          C Offline
                          C Offline
                          Creaperdown
                          wrote on last edited by Creaperdown
                          #17

                          @JonB The function is called by qml code which looks like this:

                          FileDialog
                          {
                              ...
                              onAccepted: BookController.addBook(path);
                          }
                          

                          This calls the c++ addBook method which looks like this:

                          int BookController::addBook(const QString& path)
                          {
                              auto result = m_bookService->addBook(path);
                              return static_cast<int>(result);
                          }
                          

                          Which in turn calls the next method:

                          BookOperationStatus BookService::addBook(const QString& filePath)
                          {
                             // Do some stuff....
                          
                              const Book& bookToStore = m_books.at(m_books.size() - 1);
                              m_bookStorageManager->addBook(bookToStore);  // <--- Store the book
                          
                              return BookOperationStatus::Success;
                          }
                          

                          Which calls

                          void BookStorageManager::addBook(const Book& bookToAdd)
                          {
                              // Store book to local file
                          
                              m_bookStorageGateway->createBook(m_authenticationToken, bookToAdd);  // <--- Send book to server
                          }
                          

                          Which calls:

                          void BookStorageGateway::createBook(const QString& authToken, const Book& book)
                          {
                              auto jsonDoc = QJsonDocument::fromJson(book.toJson());
                              auto jsonBook = jsonDoc.object();
                          
                              convertJsonBookToApiFormat(jsonBook);
                          
                              m_bookStorageAccess->createBook(authToken, jsonBook);
                          }
                          

                          And which then finally calls the code that I have shared before:

                          void BookStorageAccess::createBook(const QString& authToken,
                                                             const QJsonObject& jsonBook)
                          {
                             // Create multi-part
                              m_multiPart.reset(new QHttpMultiPart(QHttpMultiPart::FormDataType));
                          
                          
                              // Open file
                              QFile* file = new QFile(QUrl(jsonBook["filePath"].toString()).path());
                              if(!file->open(QIODevice::ReadOnly | QIODevice::Text))
                              {
                                  qDebug() << "Could not open test file!";
                                  return;
                              }
                          
                          
                              // Create file part
                              QHttpPart filePart;
                              filePart.setHeader(QNetworkRequest::ContentTypeHeader,
                                                 QVariant("application/pdf"));
                              filePart.setHeader(QNetworkRequest::ContentDispositionHeader,
                                                 QVariant("form-data; name=\"file\"; filename=\"" +
                                                          file->fileName() + "\""));
                          
                              filePart.setBodyDevice(file);
                              file->setParent(m_multiPart.get());
                          
                          
                              // Add file part
                              m_multiPart->append(filePart);
                          
                          
                              // Setup request
                              QUrl url(data::baseUrl + "/api/book/data");
                              QNetworkRequest testRequest { url };
                              testRequest.setRawHeader("X-Version", "1.0");
                              testRequest.setRawHeader(QByteArray("Authorization"),
                                                       "Bearer " + authToken.toUtf8());
                              testRequest.setHeader(QNetworkRequest::ContentDispositionHeader,
                                                    QVariant("form-data; name=\"file\"; filename=\"" +
                                                             file->fileName() + "\""));
                          
                          
                              // Configure SSL
                              QSslConfiguration sslConfiguration = testRequest.sslConfiguration();
                              sslConfiguration.setProtocol(QSsl::AnyProtocol);
                              sslConfiguration.setPeerVerifyMode(QSslSocket::QueryPeer);
                              testRequest.setSslConfiguration(sslConfiguration);
                          
                          
                              // Send request
                              auto reply = m_networkAccessManager.post(testRequest, m_multiPart.get());
                              m_testReply.reset(reply);
                          
                          
                              connect(m_testReply.get(), &QNetworkReply::uploadProgress,
                                      [](qint64 bytesSent, qint64 bytesTotal)
                                      {
                                          qDebug() << QString::number(bytesSent) + " of " +
                                                          QString::number(bytesTotal) + " uploaded";
                                      });
                          
                              connect(m_testReply.get(), &QNetworkReply::finished,
                                      []()
                                      {
                                          qDebug() << "Upload finished!";
                                      });
                          }
                          

                          Note that I have changed the last method a bit compared to the code that I have provided before. I tried some things, but none of them fixed my problem.

                          JonBJ 1 Reply Last reply
                          0
                          • C Creaperdown

                            @JonB The function is called by qml code which looks like this:

                            FileDialog
                            {
                                ...
                                onAccepted: BookController.addBook(path);
                            }
                            

                            This calls the c++ addBook method which looks like this:

                            int BookController::addBook(const QString& path)
                            {
                                auto result = m_bookService->addBook(path);
                                return static_cast<int>(result);
                            }
                            

                            Which in turn calls the next method:

                            BookOperationStatus BookService::addBook(const QString& filePath)
                            {
                               // Do some stuff....
                            
                                const Book& bookToStore = m_books.at(m_books.size() - 1);
                                m_bookStorageManager->addBook(bookToStore);  // <--- Store the book
                            
                                return BookOperationStatus::Success;
                            }
                            

                            Which calls

                            void BookStorageManager::addBook(const Book& bookToAdd)
                            {
                                // Store book to local file
                            
                                m_bookStorageGateway->createBook(m_authenticationToken, bookToAdd);  // <--- Send book to server
                            }
                            

                            Which calls:

                            void BookStorageGateway::createBook(const QString& authToken, const Book& book)
                            {
                                auto jsonDoc = QJsonDocument::fromJson(book.toJson());
                                auto jsonBook = jsonDoc.object();
                            
                                convertJsonBookToApiFormat(jsonBook);
                            
                                m_bookStorageAccess->createBook(authToken, jsonBook);
                            }
                            

                            And which then finally calls the code that I have shared before:

                            void BookStorageAccess::createBook(const QString& authToken,
                                                               const QJsonObject& jsonBook)
                            {
                               // Create multi-part
                                m_multiPart.reset(new QHttpMultiPart(QHttpMultiPart::FormDataType));
                            
                            
                                // Open file
                                QFile* file = new QFile(QUrl(jsonBook["filePath"].toString()).path());
                                if(!file->open(QIODevice::ReadOnly | QIODevice::Text))
                                {
                                    qDebug() << "Could not open test file!";
                                    return;
                                }
                            
                            
                                // Create file part
                                QHttpPart filePart;
                                filePart.setHeader(QNetworkRequest::ContentTypeHeader,
                                                   QVariant("application/pdf"));
                                filePart.setHeader(QNetworkRequest::ContentDispositionHeader,
                                                   QVariant("form-data; name=\"file\"; filename=\"" +
                                                            file->fileName() + "\""));
                            
                                filePart.setBodyDevice(file);
                                file->setParent(m_multiPart.get());
                            
                            
                                // Add file part
                                m_multiPart->append(filePart);
                            
                            
                                // Setup request
                                QUrl url(data::baseUrl + "/api/book/data");
                                QNetworkRequest testRequest { url };
                                testRequest.setRawHeader("X-Version", "1.0");
                                testRequest.setRawHeader(QByteArray("Authorization"),
                                                         "Bearer " + authToken.toUtf8());
                                testRequest.setHeader(QNetworkRequest::ContentDispositionHeader,
                                                      QVariant("form-data; name=\"file\"; filename=\"" +
                                                               file->fileName() + "\""));
                            
                            
                                // Configure SSL
                                QSslConfiguration sslConfiguration = testRequest.sslConfiguration();
                                sslConfiguration.setProtocol(QSsl::AnyProtocol);
                                sslConfiguration.setPeerVerifyMode(QSslSocket::QueryPeer);
                                testRequest.setSslConfiguration(sslConfiguration);
                            
                            
                                // Send request
                                auto reply = m_networkAccessManager.post(testRequest, m_multiPart.get());
                                m_testReply.reset(reply);
                            
                            
                                connect(m_testReply.get(), &QNetworkReply::uploadProgress,
                                        [](qint64 bytesSent, qint64 bytesTotal)
                                        {
                                            qDebug() << QString::number(bytesSent) + " of " +
                                                            QString::number(bytesTotal) + " uploaded";
                                        });
                            
                                connect(m_testReply.get(), &QNetworkReply::finished,
                                        []()
                                        {
                                            qDebug() << "Upload finished!";
                                        });
                            }
                            

                            Note that I have changed the last method a bit compared to the code that I have provided before. I tried some things, but none of them fixed my problem.

                            JonBJ Offline
                            JonBJ Offline
                            JonB
                            wrote on last edited by JonB
                            #18

                            @Creaperdown
                            Firstly this is the first time you have mentioned you are using QML. It's probably fine, but I know nothing about QML.

                            This may all be a rabbit hole, but I don't know what your m_multiPart & m_testReply are/how they work. From my knowledge of C++ I don't know how you get

                            m_multiPart.get()
                            // together with
                            m_multiPart->append(filePart)
                            

                            to work (compile). [Maybe it works if get() is a static method of YourDerivedHttpMultiPart so equivalent to YourDerivedHttpMultiPart::get(), I don't know.] The important thing is that necessary objects have a lifetime across the file transfer. If I wanted to be sure I might put connect(object, &QObject::destroyed, this, [object]() { qDebug() << object << " destroyed"; } ) on those objects derived from QObject you create/use to make sure I knew when they get destroyed. If they are somehow "short-lived" maybe it's possible they send the first "block" of the transfer but no more.

                            C 2 Replies Last reply
                            0
                            • JonBJ JonB

                              @Creaperdown
                              Firstly this is the first time you have mentioned you are using QML. It's probably fine, but I know nothing about QML.

                              This may all be a rabbit hole, but I don't know what your m_multiPart & m_testReply are/how they work. From my knowledge of C++ I don't know how you get

                              m_multiPart.get()
                              // together with
                              m_multiPart->append(filePart)
                              

                              to work (compile). [Maybe it works if get() is a static method of YourDerivedHttpMultiPart so equivalent to YourDerivedHttpMultiPart::get(), I don't know.] The important thing is that necessary objects have a lifetime across the file transfer. If I wanted to be sure I might put connect(object, &QObject::destroyed, this, [object]() { qDebug() << object << " destroyed"; } ) on those objects derived from QObject you create/use to make sure I knew when they get destroyed. If they are somehow "short-lived" maybe it's possible they send the first "block" of the transfer but no more.

                              C Offline
                              C Offline
                              Creaperdown
                              wrote on last edited by
                              #19

                              @JonB m_multiPart is a std::unique_ptr<QHttpMultiPart> in the header file of my class. I'll now try connecting a slot to the destroyed signal of QObject to know when they are destroyed

                              1 Reply Last reply
                              0
                              • JonBJ JonB

                                @Creaperdown
                                Firstly this is the first time you have mentioned you are using QML. It's probably fine, but I know nothing about QML.

                                This may all be a rabbit hole, but I don't know what your m_multiPart & m_testReply are/how they work. From my knowledge of C++ I don't know how you get

                                m_multiPart.get()
                                // together with
                                m_multiPart->append(filePart)
                                

                                to work (compile). [Maybe it works if get() is a static method of YourDerivedHttpMultiPart so equivalent to YourDerivedHttpMultiPart::get(), I don't know.] The important thing is that necessary objects have a lifetime across the file transfer. If I wanted to be sure I might put connect(object, &QObject::destroyed, this, [object]() { qDebug() << object << " destroyed"; } ) on those objects derived from QObject you create/use to make sure I knew when they get destroyed. If they are somehow "short-lived" maybe it's possible they send the first "block" of the transfer but no more.

                                C Offline
                                C Offline
                                Creaperdown
                                wrote on last edited by
                                #20

                                @JonB I have now added:

                                // Create multi-part
                                    m_multiPart.reset(new QHttpMultiPart(QHttpMultiPart::FormDataType));
                                    connect(m_multiPart.get(), &QObject::destroyed,
                                            []()
                                            {
                                                qDebug() << "multipart destroyed!";
                                            });
                                
                                
                                    // Open file
                                    QFile* file = new QFile(QUrl(jsonBook["filePath"].toString()).path());
                                    if(!file->open(QIODevice::ReadOnly | QIODevice::Text))
                                    {
                                        qDebug() << "Could not open test file!";
                                        return;
                                    }
                                    connect(file, &QObject::destroyed,
                                            []()
                                            {
                                                qDebug() << "file destroyed!";
                                            });
                                

                                But nothing is ever printed, thus the objects should never get destroyed

                                1 Reply Last reply
                                0
                                • C Offline
                                  C Offline
                                  Creaperdown
                                  wrote on last edited by
                                  #21

                                  I have found the problem...

                                  I have called: file->open(QIODevice::ReadOnly | QIODevice::Text) while the file I am reading is not Text, opening the file simply via: file->open(QIODevice::ReadOnly) fixed it.

                                  1 Reply Last reply
                                  0
                                  • C Creaperdown has marked this topic as solved on
                                  • JonBJ JonB referenced this topic on

                                  • Login

                                  • Login or register to search.
                                  • First post
                                    Last post
                                  0
                                  • Categories
                                  • Recent
                                  • Tags
                                  • Popular
                                  • Users
                                  • Groups
                                  • Search
                                  • Get Qt Extensions
                                  • Unsolved