Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Not able to transfer large files on QTcpSocket
Forum Updated to NodeBB v4.3 + New Features

Not able to transfer large files on QTcpSocket

Scheduled Pinned Locked Moved Unsolved General and Desktop
4 Posts 3 Posters 676 Views 1 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • dualD Offline
    dualD Offline
    dual
    wrote on last edited by dual
    #1

    Hi everybody, I'm trying to code a server-client application that automatically transfers files from a client to a server but I'm not able to send files largen than 8-10 Mb to the server.
    On Windows it seems to work well, in fact I'm able to transfer files with size of more or less 5 GB, but on unix/macOS I cannot: the readFile function triggers only few times and doesn't receive the whole transmitted bytes.

    This is the code:
    Client-side:

    void client::sendFile(std::string path_to_watch) {
        file = new QFile(QString::fromStdString(path_to_watch));
        if (!file->open(QIODevice::ReadWrite))
        {
            qDebug()<<"Couldn't open the file";
            return;
        }
        qint64 file_size = file->size();
    
        QFileInfo fileInfo(file->fileName());
        QJsonObject file_created{
                {"action", 1},
                {"file_name", fileInfo.fileName()},
                {"file_size", file_size}
        };
    
        t->sendJson(file_created, address, 50505);
    
        QByteArray block; 
    
        connect(t->getTcpSocket(), SIGNAL(disconnected()),
                t->getTcpSocket(), SLOT(deleteLater()));
    
        qint64 sentData = 0;
        qint64 leftData = file->size()-sentData;
        while(leftData>0){
            if(leftData > 100000000){
                block = file->read(100000000);
            }else{
                block = file->read(leftData);
            }
            qint64 written = t->getTcpSocket()->write(block);
            sentData+=written;
            leftData = file->size()-sentData;
            t->getTcpSocket()->waitForBytesWritten();
            block.remove(0, block.size());
        }
        file->close();
        return;
    }
    

    Server-side:

    void server::receive(){
    
        in.startTransaction();
    
        QString nextFortune;
        in >> nextFortune;
        QJsonDocument jsonResponse = QJsonDocument::fromJson(nextFortune.toLatin1());
        QJsonArray jsonArray = jsonResponse.array();
    
        if(!jsonArray.isEmpty())   
        {
            QJsonObject jsonObject = jsonArray.first().toObject();
    
            if (!in.commitTransaction())
                return;
    
            int c = jsonObject.value("action").toInt();                
            QJsonArray jsarray; 
    
            switch (c) {
    ...
                case 1: {
                    std::cout<<"entro in switch 1..."<<std::endl;
                    file_name = jsonObject.value("file_name").toString();
                    file_size = jsonObject.value("file_size").toVariant().toLongLong();
    
                    disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
                    connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
                }
            break;
    
    ...
            }
        }
    
    }
    
    
    void server::readFile(){
        if(firstTry){
            QByteArray b;
            blocks.append(b);
            total=0;
            firstTry = false;
        }
    
        if(blocks.last().size() > 1400000000){
            QByteArray b1;
            blocks.append(b1);
        }
    
        if(total < file_size){
            qint64 available = clientConnection->bytesAvailable();
            total += available;
            std::cout<<"Available: "<<available<<std::endl;
            std::cout<<"Total: "<<total << std::endl;
            blocks.last().append(clientConnection->read(available));
            if(total<file_size){
                return;
            }else{
                firstTry=true;
                dataReceived();
            }
        }
    }
    

    blocks is a QVector of QByteArray but it's temporary, I'll change it as soon as I'm able to transmit the files correctly
    Any idea?
    Thank you in advance

    JonBJ 1 Reply Last reply
    0
    • dualD dual

      Hi everybody, I'm trying to code a server-client application that automatically transfers files from a client to a server but I'm not able to send files largen than 8-10 Mb to the server.
      On Windows it seems to work well, in fact I'm able to transfer files with size of more or less 5 GB, but on unix/macOS I cannot: the readFile function triggers only few times and doesn't receive the whole transmitted bytes.

      This is the code:
      Client-side:

      void client::sendFile(std::string path_to_watch) {
          file = new QFile(QString::fromStdString(path_to_watch));
          if (!file->open(QIODevice::ReadWrite))
          {
              qDebug()<<"Couldn't open the file";
              return;
          }
          qint64 file_size = file->size();
      
          QFileInfo fileInfo(file->fileName());
          QJsonObject file_created{
                  {"action", 1},
                  {"file_name", fileInfo.fileName()},
                  {"file_size", file_size}
          };
      
          t->sendJson(file_created, address, 50505);
      
          QByteArray block; 
      
          connect(t->getTcpSocket(), SIGNAL(disconnected()),
                  t->getTcpSocket(), SLOT(deleteLater()));
      
          qint64 sentData = 0;
          qint64 leftData = file->size()-sentData;
          while(leftData>0){
              if(leftData > 100000000){
                  block = file->read(100000000);
              }else{
                  block = file->read(leftData);
              }
              qint64 written = t->getTcpSocket()->write(block);
              sentData+=written;
              leftData = file->size()-sentData;
              t->getTcpSocket()->waitForBytesWritten();
              block.remove(0, block.size());
          }
          file->close();
          return;
      }
      

      Server-side:

      void server::receive(){
      
          in.startTransaction();
      
          QString nextFortune;
          in >> nextFortune;
          QJsonDocument jsonResponse = QJsonDocument::fromJson(nextFortune.toLatin1());
          QJsonArray jsonArray = jsonResponse.array();
      
          if(!jsonArray.isEmpty())   
          {
              QJsonObject jsonObject = jsonArray.first().toObject();
      
              if (!in.commitTransaction())
                  return;
      
              int c = jsonObject.value("action").toInt();                
              QJsonArray jsarray; 
      
              switch (c) {
      ...
                  case 1: {
                      std::cout<<"entro in switch 1..."<<std::endl;
                      file_name = jsonObject.value("file_name").toString();
                      file_size = jsonObject.value("file_size").toVariant().toLongLong();
      
                      disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
                      connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
                  }
              break;
      
      ...
              }
          }
      
      }
      
      
      void server::readFile(){
          if(firstTry){
              QByteArray b;
              blocks.append(b);
              total=0;
              firstTry = false;
          }
      
          if(blocks.last().size() > 1400000000){
              QByteArray b1;
              blocks.append(b1);
          }
      
          if(total < file_size){
              qint64 available = clientConnection->bytesAvailable();
              total += available;
              std::cout<<"Available: "<<available<<std::endl;
              std::cout<<"Total: "<<total << std::endl;
              blocks.last().append(clientConnection->read(available));
              if(total<file_size){
                  return;
              }else{
                  firstTry=true;
                  dataReceived();
              }
          }
      }
      

      blocks is a QVector of QByteArray but it's temporary, I'll change it as soon as I'm able to transmit the files correctly
      Any idea?
      Thank you in advance

      JonBJ Offline
      JonBJ Offline
      JonB
      wrote on last edited by
      #2

      @dual
      Not sure, because you have a heck of a lot of code and no comments/debug information.

      Your code looks incredibly complex. It should be miles shorter. I wouldn't expect to see any waitFor...()s, bytesAvailable(), firstTrys, or half the other stuff. I am not sure about your transaction either. There is lots of code missing. I presume you are compiling 64-bit, and all your relevant variables used here are correctly 64-bit (e.g. not plain int).

      disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
      connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
      

      I don't like the look of the above. You are switching slots while a transfer is underway. The client sends a stream of data, you cannot assume you receive them in "boundaries" which have any match to what calls you used to send them. I am concerned that by the time you start getting readFile() called you may have already received some of the bytes. (Maybe your transaction prevents that, I don't know.)

      If it were me, I would start again and make the code much simpler...! :)

      1 Reply Last reply
      1
      • dualD Offline
        dualD Offline
        dual
        wrote on last edited by dual
        #3

        @JonB
        Thanks for your answer.
        My code became complex because I tried to make it work on Windows machines at least.
        But let me explain some choices I made:

        disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
        connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
        

        I used these two functions because I need to differentiate the receiving of a Json (which provides the different commands -like login, sign up, sign out, infos about the file size, etc-) from the receiving of the file itself.

                qint64 available = clientConnection->bytesAvailable();
        

        This function was necessary because readAll()/read() will stop working when receiving a large sized file. I think here there's room for improvement, in fact I accept any suggestion in order to manage this issue.

        if(firstTry){
                ...
            }
        

        I'm using the boolean firstTry in order to initialize the data struct I will use for receiving the file, in fact the readFile will be called many times and I need to initialize it only once

        1 Reply Last reply
        0
        • SGaistS Offline
          SGaistS Offline
          SGaist
          Lifetime Qt Champion
          wrote on last edited by
          #4

          Hi,

          From the looks of it you have a protocol to follow. You should implement that logic properly rather than relying on some hazardous state.
          It looks like you should have something along the line of:

          • send json data
          • get ack from server
          • send file
          • get ack from server

          Rince and repeat.

          You might want to take a look at Amazon's S3 chunked API. It has a pretty good design and implements a pretty nice way to upload heavy files.

          Interested in AI ? www.idiap.ch
          Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

          1 Reply Last reply
          3

          • Login

          • Login or register to search.
          • First post
            Last post
          0
          • Categories
          • Recent
          • Tags
          • Popular
          • Users
          • Groups
          • Search
          • Get Qt Extensions
          • Unsolved