Sending Large file over QTcpSocket

[SlayH]

code_text
```Please Anyone, Help!!!!!
I receive the file but it is empty. I don't know whether the error occurs while dividing the file into chunks or when reconstructing it to save it.  

Below are the function related to sending the file.

void divide_file_into_packets(QString file_name)
{
    QFile file(file_name);
    if (file.open(QIODevice::ReadOnly))
    {
        qint64 sequence_number = 0;

        while (!file.atEnd())
        {
            QByteArray packet;
            QDataStream out(&packet, QIODevice::ReadWrite);
            out.setVersion(QDataStream::Qt_6_0);

            qint64 start_position = file.pos();    // Record the start position of the packet
            out << send_file << sequence_number++; // Add sequence number to the packet

            // Read data from the file directly into the packet
            packet.append(file.read(PACKET_SIZE - sizeof(qint64)));

            // Move the file pointer to the next position for the next packet
            file.seek(start_position + packet.size() - sizeof(qint64));

            _packets.append(packet); // Store packet
        }
    }

    file.close();
}

void send_file()
{

    _protocol->divide_file_into_packets(_file_name);

    for (QByteArray packet : _protocol->packets())
        _socket->write(packet);
}

void load_data(QList<QByteArray> data)
{
    for (QByteArray packet : data)
    {
        QDataStream in(&packet, QIODevice::ReadWrite);
        in.setVersion(QDataStream::Qt_6_0);

        in >> _type;

        qint64 sequence_number = 0;
        QByteArray temp;

        switch (_type)
        {
        case text:
            in >> _message;

            break;

        case set_name:
            in >> _name;

            break;

        case is_typing:
            in >> _is_typing;

            break;

        case init_sending_file:
            in >> _file_name >> _file_size;

            break;

        case send_file:
            in >> sequence_number >> temp;

            _file_data_map[sequence_number] = temp;

            break;

        default:
            break;
        }
    }
}

void ready_read()
{
    QList<QByteArray> data;
    while (_socket->bytesAvailable())
        data << _socket->readAll();

    _protocol->load_data(data);

    switch (_protocol->type())
    {
    case text:
        emit text_message_received(_protocol->message());

        break;

    case is_typing:
        emit is_typing_received();

        break;

    case init_sending_file:
        emit init_receiving_file(_protocol->name(), _protocol->file_name(), _protocol->file_size());

        break;

    case accept_sending_file:
        send_file();

        break;

    case reject_sending_file:
        emit reject_receiving_file();

        break;

    case send_file:
        save_file();

    default:
        break;
    }
}

void save_file()
{
    QDir dir;
    dir.mkdir("save_folder");
    dir.setPath("./");

    QString path = QString("%1%2").arg(dir.canonicalPath(), name());

    QFile file(path);

    if (file.open(QIODevice::WriteOnly))
    {
        const QMap<qint64, QByteArray> &fileDataMap = _protocol->file_data_map();

        for (auto it = fileDataMap.constBegin(); it != fileDataMap.constEnd(); it++)
            file.write(it.value());

        file.close();

        emit file_saved(path);
    }
}

[JonB]

@SlayH
Please edit your code to use the forum's Code tags for code blocks. And preferably reduce to a minimal example.

[JonB]

@SlayH
I see QDataStream being used when loading data. Where is it being used when sending data? (Oh I think I can see that.)

    while (_socket->bytesAvailable())
        data << _socket->readAll();

    _protocol->load_data(data);

How does load_data() cope with "partial" data? If you think that your socket reading loop loads individual packets (or all data packets) it does not.

I receive the file but it is empty.

Put debug statements in so that you show us (and yourself) what is happening when receiving.

[SlayH]

@JonB The client does receive it in chunks cause my packets are 100 bytes and I just sent a file of 150 and it did send 2 packets. This is the statement printed in the terminal.

╭─    ~/Documents/server---clients/build/clients/client.app/Contents/MacOS    main !19 ?5 ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── SEGV ✘  51s   10:25:45 PM  ─╮
╰─ /Users/test/Documents/server---clients/build/clients/client.app/Contents/MacOS/client ─╯
2024-03-23 22:26:40.916 client[33849:346985] TSM AdjustCapsLockLEDForKeyTransitionHandling - _ISSetPhysicalKeyboardCapsLockLED Inhibit
Data received
Data received

This is the modified code.

code_text```
code_text

QList<QByteArray> data;
    while (_socket->bytesAvailable())
    {
        data << _socket->readAll();
        qDebug() << "Data received";
    }

    _protocol->load_data(data);```
code_text

[Ronel_qtmaster]

@SlayH You have two ways of doing it.
the quickest way is to use Qnetworkaccessmanager::put function to a tcpserver.You can refer to a project i did as well https://github.com/jordanprog86/FilesTransfer.git

the second way is to use QTcpSocket.
You send all the data to the server.
Then from the server you write progressively the received data to the file untill it stops sending

[JonB]

@JonB said in Sending Large file over QTcpSocket:

How does load_data() cope with "partial" data? If you think that your socket reading loop loads individual packets (or all data packets) it does not.

You do not cope with this. You send a bunch of packets. The receiver just receives a stream of bytes. You have no idea when readyRead() gets called But you send no indication of how many packets there are. So the receiver has no idea and how many packets to wait for for a complete file, as it might do if e.g. you sent it the final value of sequence_number.

Your debugging should go into load_data() to see how many times it is called, with what data. You should see how many times you are calling save_file(), which completely overwrites the output file silently.

Additional as a BTW, I don't see what the whole splitting of the sent file into chunks achieves at all. The only thing you put in each packet is a number plus a fixed PACKET_SIZE of bytes. You might just as well send the whole file in one call preceded by a total byte count.

[JonB]

@Ronel_qtmaster said in Sending Large file over QTcpSocket:

Then from the server you write progressively the received data to the file untill it stops sending

And how would that know "when it stops sending"? To send e.g. a file you need some kind of "total bytes" or equivalent, or a unique "end of file" marker, in the protocol you stream.

[SlayH]

@JonB Yeah You are right. I should have embedded the stream with an end file marker or how many packets there are. Gonna try to do it now.

[Ronel_qtmaster]

@JonB that is why i suggested upload as the first solution because it uses a boundary at the beginning
@SlayH this is an exemple for uploading an image and text

QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);

QHttpPart textPart;
textPart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name="text""));
textPart.setBody("my text");

QHttpPart imagePart;
imagePart.setHeader(QNetworkRequest::ContentTypeHeader, QVariant("image/jpeg"));
imagePart.setHeader(QNetworkRequest::ContentDispositionHeader, QVariant("form-data; name="image""));
QFile *file = new QFile("image.jpg");
file->open(QIODevice::ReadOnly);
imagePart.setBodyDevice(file);
file->setParent(multiPart); // we cannot delete the file now, so delete it with the multiPart

multiPart->append(textPart);
multiPart->append(imagePart);

QUrl url("http://my.server.tld");
QNetworkRequest request(url);

QNetworkAccessManager manager;
QNetworkReply *reply = manager.post(request, multiPart);
multiPart->setParent(reply); // delete the multiPart with the reply

[JonB]

@SlayH said in Sending Large file over QTcpSocket:

with an end file marker

How are you going to do that when sending packets which contain any pattern of binary bytes read from a file? I think you will need to precede by a byte count (or packet count or some kind of "special/empty" packet at the endif you really wish, but as I said I don't think your packet-ization achieves anything at all or what you might think it achieves).

Also, as I said earlier. I assume your ready_read() is (supposed to be) a slot for readyRead() signal? Then as long as you have a pattern like

    while (_socket->bytesAvailable())
        data << _socket->readAll();

and assume (as you do) that data will be some kind of "complete set of data or packets" then your algorithm is fatally flawed. Might work for a bit/some cases, might not. You are assuming a relationship which does not exist, such as that for each _socket->write(packet) you will receive some single read, which is not at all true. This is partly why giving up on packets and just sending a byte count followed by a stream of binary bytes would be simpler.

[JonB]

@Ronel_qtmaster
What makes you think the OP has any kind of HTTP server here? I don't see that he has indicated any such thing?

[Ronel_qtmaster]

@JonB it is just an exemple.If he has a QTcpServer listening to 127.0.01 and port 455 for exemple, he can change the url to QUrl("Http://127.0.0.1:455"), which will reach the tcpserver listening

[JonB]

@Ronel_qtmaster
And this QTcpServer will be able to service HTTP requests, so that you can send your file with QHttpMultiPart which is what you are advocating? Without any Q[Abstract]HttpServer? I must be misunderstanding something then.

[SlayH]

@JonB Yeah ready_read() is a slot for readyRead(). I did think that for each _socket->write(packet) there will be an emitted signal for readRead() but You told me it isn't the case.``` If I am supposed to send a byte count followed by a stream of binary bytes, how would it be then ???
code_text

[JonB]

@SlayH said in Sending Large file over QTcpSocket:

I did think that for each _socket->write(packet) there will be an emitted signal for readRead()

That's what I tried to say at the outset, because so many people assume it. It is not the case, at all sorts of levels in the handling.

I'm not going to write full code. In outline all you have to do is

int size = ...;
sendsocket->write(&size, sizeof(size));
sendsocket->write(fileContent, size);

at sender side.

At receiver side, read that initial size and then keep writing received bytes to file till that many bytes have been read. All reads go through onReadyRead slot, but you can only assume that will receive 1 or more bytes any time it's called. You need to receive at least sizeof(size) bytes (allow for accumulating that many bytes over multiple calls to onReadyRead() in e.g. a class member buffer), all subsequent bytes can go to output file counting up/down to that many.

You may want to think about how to send that int (do you care about byte-order for cross-platform?). You may find QDataStream's Using Read Transactions useful if you don't want to write your own stuff. (You may only want to use that for the leading int. For the "potentially large" amount of data which follows for the file content you will not want to read all of that into memory/a variable, rather only "chunks" at a time from whatever happens to have arrived there, immediately copied out to destination file.)

[SlayH]

@JonB Thanks. I'm gonna try it.

[SGaist]

Hi,

Since you are using QDataStream, you can use transactions.

[Ronel_qtmaster]

@JonB Yes exactly.The only thing is that he will have to remove the request headers from the sent packet, and put the original data in a file.

In a nutshell, when sending files with upload, the request headers are sent with the file data.You just need to separate both and keep the essential part