Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

Not able to transfer large files on QTcpSocket



  • Hi everybody, I'm trying to code a server-client application that automatically transfers files from a client to a server but I'm not able to send files largen than 8-10 Mb to the server.
    On Windows it seems to work well, in fact I'm able to transfer files with size of more or less 5 GB, but on unix/macOS I cannot: the readFile function triggers only few times and doesn't receive the whole transmitted bytes.

    This is the code:
    Client-side:

    void client::sendFile(std::string path_to_watch) {
        file = new QFile(QString::fromStdString(path_to_watch));
        if (!file->open(QIODevice::ReadWrite))
        {
            qDebug()<<"Couldn't open the file";
            return;
        }
        qint64 file_size = file->size();
    
        QFileInfo fileInfo(file->fileName());
        QJsonObject file_created{
                {"action", 1},
                {"file_name", fileInfo.fileName()},
                {"file_size", file_size}
        };
    
        t->sendJson(file_created, address, 50505);
    
        QByteArray block; 
    
        connect(t->getTcpSocket(), SIGNAL(disconnected()),
                t->getTcpSocket(), SLOT(deleteLater()));
    
        qint64 sentData = 0;
        qint64 leftData = file->size()-sentData;
        while(leftData>0){
            if(leftData > 100000000){
                block = file->read(100000000);
            }else{
                block = file->read(leftData);
            }
            qint64 written = t->getTcpSocket()->write(block);
            sentData+=written;
            leftData = file->size()-sentData;
            t->getTcpSocket()->waitForBytesWritten();
            block.remove(0, block.size());
        }
        file->close();
        return;
    }
    

    Server-side:

    void server::receive(){
    
        in.startTransaction();
    
        QString nextFortune;
        in >> nextFortune;
        QJsonDocument jsonResponse = QJsonDocument::fromJson(nextFortune.toLatin1());
        QJsonArray jsonArray = jsonResponse.array();
    
        if(!jsonArray.isEmpty())   
        {
            QJsonObject jsonObject = jsonArray.first().toObject();
    
            if (!in.commitTransaction())
                return;
    
            int c = jsonObject.value("action").toInt();                
            QJsonArray jsarray; 
    
            switch (c) {
    ...
                case 1: {
                    std::cout<<"entro in switch 1..."<<std::endl;
                    file_name = jsonObject.value("file_name").toString();
                    file_size = jsonObject.value("file_size").toVariant().toLongLong();
    
                    disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
                    connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
                }
            break;
    
    ...
            }
        }
    
    }
    
    
    void server::readFile(){
        if(firstTry){
            QByteArray b;
            blocks.append(b);
            total=0;
            firstTry = false;
        }
    
        if(blocks.last().size() > 1400000000){
            QByteArray b1;
            blocks.append(b1);
        }
    
        if(total < file_size){
            qint64 available = clientConnection->bytesAvailable();
            total += available;
            std::cout<<"Available: "<<available<<std::endl;
            std::cout<<"Total: "<<total << std::endl;
            blocks.last().append(clientConnection->read(available));
            if(total<file_size){
                return;
            }else{
                firstTry=true;
                dataReceived();
            }
        }
    }
    

    blocks is a QVector of QByteArray but it's temporary, I'll change it as soon as I'm able to transmit the files correctly
    Any idea?
    Thank you in advance



  • @dual
    Not sure, because you have a heck of a lot of code and no comments/debug information.

    Your code looks incredibly complex. It should be miles shorter. I wouldn't expect to see any waitFor...()s, bytesAvailable(), firstTrys, or half the other stuff. I am not sure about your transaction either. There is lots of code missing. I presume you are compiling 64-bit, and all your relevant variables used here are correctly 64-bit (e.g. not plain int).

    disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
    connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
    

    I don't like the look of the above. You are switching slots while a transfer is underway. The client sends a stream of data, you cannot assume you receive them in "boundaries" which have any match to what calls you used to send them. I am concerned that by the time you start getting readFile() called you may have already received some of the bytes. (Maybe your transaction prevents that, I don't know.)

    If it were me, I would start again and make the code much simpler...! :)



  • @JonB
    Thanks for your answer.
    My code became complex because I tried to make it work on Windows machines at least.
    But let me explain some choices I made:

    disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
    connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
    

    I used these two functions because I need to differentiate the receiving of a Json (which provides the different commands -like login, sign up, sign out, infos about the file size, etc-) from the receiving of the file itself.

            qint64 available = clientConnection->bytesAvailable();
    

    This function was necessary because readAll()/read() will stop working when receiving a large sized file. I think here there's room for improvement, in fact I accept any suggestion in order to manage this issue.

    if(firstTry){
            ...
        }
    

    I'm using the boolean firstTry in order to initialize the data struct I will use for receiving the file, in fact the readFile will be called many times and I need to initialize it only once


  • Lifetime Qt Champion

    Hi,

    From the looks of it you have a protocol to follow. You should implement that logic properly rather than relying on some hazardous state.
    It looks like you should have something along the line of:

    • send json data
    • get ack from server
    • send file
    • get ack from server

    Rince and repeat.

    You might want to take a look at Amazon's S3 chunked API. It has a pretty good design and implements a pretty nice way to upload heavy files.


Log in to reply