Not able to transfer large files on QTcpSocket

[dual]

Hi everybody, I'm trying to code a server-client application that automatically transfers files from a client to a server but I'm not able to send files largen than 8-10 Mb to the server.
On Windows it seems to work well, in fact I'm able to transfer files with size of more or less 5 GB, but on unix/macOS I cannot: the readFile function triggers only few times and doesn't receive the whole transmitted bytes.

This is the code:
Client-side:

void client::sendFile(std::string path_to_watch) {
    file = new QFile(QString::fromStdString(path_to_watch));
    if (!file->open(QIODevice::ReadWrite))
    {
        qDebug()<<"Couldn't open the file";
        return;
    }
    qint64 file_size = file->size();

    QFileInfo fileInfo(file->fileName());
    QJsonObject file_created{
            {"action", 1},
            {"file_name", fileInfo.fileName()},
            {"file_size", file_size}
    };

    t->sendJson(file_created, address, 50505);

    QByteArray block; 

    connect(t->getTcpSocket(), SIGNAL(disconnected()),
            t->getTcpSocket(), SLOT(deleteLater()));

    qint64 sentData = 0;
    qint64 leftData = file->size()-sentData;
    while(leftData>0){
        if(leftData > 100000000){
            block = file->read(100000000);
        }else{
            block = file->read(leftData);
        }
        qint64 written = t->getTcpSocket()->write(block);
        sentData+=written;
        leftData = file->size()-sentData;
        t->getTcpSocket()->waitForBytesWritten();
        block.remove(0, block.size());
    }
    file->close();
    return;
}

Server-side:

void server::receive(){

    in.startTransaction();

    QString nextFortune;
    in >> nextFortune;
    QJsonDocument jsonResponse = QJsonDocument::fromJson(nextFortune.toLatin1());
    QJsonArray jsonArray = jsonResponse.array();

    if(!jsonArray.isEmpty())   
    {
        QJsonObject jsonObject = jsonArray.first().toObject();

        if (!in.commitTransaction())
            return;

        int c = jsonObject.value("action").toInt();                
        QJsonArray jsarray; 

        switch (c) {
...
            case 1: {
                std::cout<<"entro in switch 1..."<<std::endl;
                file_name = jsonObject.value("file_name").toString();
                file_size = jsonObject.value("file_size").toVariant().toLongLong();

                disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
                connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
            }
        break;

...
        }
    }

}


void server::readFile(){
    if(firstTry){
        QByteArray b;
        blocks.append(b);
        total=0;
        firstTry = false;
    }

    if(blocks.last().size() > 1400000000){
        QByteArray b1;
        blocks.append(b1);
    }

    if(total < file_size){
        qint64 available = clientConnection->bytesAvailable();
        total += available;
        std::cout<<"Available: "<<available<<std::endl;
        std::cout<<"Total: "<<total << std::endl;
        blocks.last().append(clientConnection->read(available));
        if(total<file_size){
            return;
        }else{
            firstTry=true;
            dataReceived();
        }
    }
}

blocks is a QVector of QByteArray but it's temporary, I'll change it as soon as I'm able to transmit the files correctly
Any idea?
Thank you in advance

[JonB]

@dual
Not sure, because you have a heck of a lot of code and no comments/debug information.

Your code looks incredibly complex. It should be miles shorter. I wouldn't expect to see any waitFor...()s, bytesAvailable(), firstTrys, or half the other stuff. I am not sure about your transaction either. There is lots of code missing. I presume you are compiling 64-bit, and all your relevant variables used here are correctly 64-bit (e.g. not plain int).

disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);

I don't like the look of the above. You are switching slots while a transfer is underway. The client sends a stream of data, you cannot assume you receive them in "boundaries" which have any match to what calls you used to send them. I am concerned that by the time you start getting readFile() called you may have already received some of the bytes. (Maybe your transaction prevents that, I don't know.)

If it were me, I would start again and make the code much simpler...! :)

[dual]

@JonB
Thanks for your answer.
My code became complex because I tried to make it work on Windows machines at least.
But let me explain some choices I made:

disconnect(clientConnection, &QIODevice::readyRead, 0, 0);
connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);

I used these two functions because I need to differentiate the receiving of a Json (which provides the different commands -like login, sign up, sign out, infos about the file size, etc-) from the receiving of the file itself.

        qint64 available = clientConnection->bytesAvailable();

This function was necessary because readAll()/read() will stop working when receiving a large sized file. I think here there's room for improvement, in fact I accept any suggestion in order to manage this issue.

if(firstTry){
        ...
    }

I'm using the boolean firstTry in order to initialize the data struct I will use for receiving the file, in fact the readFile will be called many times and I need to initialize it only once

[SGaist]

Hi,

From the looks of it you have a protocol to follow. You should implement that logic properly rather than relying on some hazardous state.
It looks like you should have something along the line of:

• send json data
• get ack from server
• send file
• get ack from server

Rince and repeat.

You might want to take a look at Amazon's S3 chunked API. It has a pretty good design and implements a pretty nice way to upload heavy files.