[QTcpSocket - Qt-6.2.7] Write to socket big raw data

nico88desmo

Dear all,

I need to send ~180MB of raw data to a device using a QTcpSocket.

Here is the code used for this function:

bool sendData(const char* stream, quint32 streamSizeBytes, quint32 timeoutMs) {

    quint32 bytesWritten = 0;

    while (bytesWritten < streamSizeBytes) {

        // **** bytesWritten here return 180 MB always ****
        bytesWritten = socket.write(stream + bytesWritten, streamSizeBytes - bytesWritten); // (1)

        if (bytesWritten == -1)
            return false;

        quint32 bytesChunkWritten = 0;
        quint32 bytesToWrite = bytesWritten;

        while (bytesToWrite > 0) {

            bool isDataWritten = socket.waitForBytesWritten(timeoutMs); // (2)

            // **** bytesToWrite here has the actual remaining bytes to send ****
            bytesToWrite = socket.bytesToWrite(); // (3)

            bytesChunkWritten = bytesWritten - bytesToWrite;

            if (bytesChunkWritten == -1 || !isDataWritten)
                return false;
        }
    }
    return true;
}

I have two doubts:

supposing streamSizeBytes is 180 * 1024 * 1024 = 188743680 bytes, once I call this method (1), effectivetly socket.write(..) returns 188743680 as written bytes; which is the meaning of this value? Qt QIODevice docs say that is the maxSize number of bytes sent from data to device; for QTcpSocket, is this the buffer area available for socket writing?
I'm using a synch (2) approach for data sending (using waitForBytesWritten(..)); as I correctly see, data are sent once I call this method; however, the real bytes size sent to the socket isnt't 180 MB, but it is much lower; in fact, the method (3) socket.bytesToWrite() confirm this, i.e. I need to iterate until bytesToWrite reach 0.

From what I'm seeing, I'm thinking the following: socket.write() returns maxSize bytes to send, but effectivetly this byte data may be split onto different chunks which size may depend to destination buffer's size; that's why I have to check how many data have been sent using socket.bytesToWrite(). May be this caused due to QTcpSocket is using a buffer area, due to its reliable connection?

For completeness, here is dev environment:

Ubuntu 22.04 LTS 64bit
Qt-6.2.7
g++-11

Thanks in advance
Nicola

JonB

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

This is my doubt: socket.write() returns that I (theorically) sent 180 MB (that is why I write that comment), but this isn't true: in practice, it seems the value return from socket.write() doesn't represent the real data sent but, instead, the data that are "scheduled" to be sent.

Yes if you want to call it that it's what is "scheduled" to be sent, in that write() does not block until all the data is physically sent. Why should it? This is also physically true if, say, you write to a file instead of a socket. Actual commit to disk happens in the background over time. You only slow the OS down/block your calling program if you do, say, an fsync(). OSes buffer stuff for your benefit all over the place. The data is written whether you call waitForBytesWritten() or bytesToWrite() or not, so you have not explained why you wish to call them. If you want to that's OK, but I don't see the point much.

At the beginning, I don't use any loop to check how many data have been sent (cause socket.write() returns 180 MB), but in practice, this isn't correct.

What is not "correct"? You are only choosing to call it "not correct" because it does not tell you how many bytes have been physically sent at that point, so what/why do you care?

JonB

@nico88desmo
TCP stuff is physically sent in "chunks", nowhere near 180MB. Furthermore, the OS may well impose its own buffering/flushing, outside of Qt's knowledge. You should not write, or need to write, code which cares about this. You should just regard as a "continuous stream", any number of bytes may be written/arrive at any time. I'm not sure why you need to use waitForBytesWritten() at all.

artwaw

@nico88desmo To add: if you don't need any sophisticated control and just want to send data, you can use QDataStream with a socket. All the data is buffered and send under to OS control.

Please be aware that the usual flush control is implemented into the socket: should you call disconnectFromHost() (the right way to close the connection) the buffer is sent out and only after then the connection is closed. This may introduce some waiting time.

nico88desmo

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

@nico88desmo
TCP stuff is physically sent in "chunks", nowhere near 180MB. Furthermore, the OS may well impose its own buffering/flushing, outside of Qt's knowledge. You should not write, or need to write, code which cares about this. You should just regard as a "continuous stream", any number of bytes may be written/arrive at any time. I'm not sure why you need to use waitForBytesWritten() at all.

Thanks for the answer; each time I call waitForBytesWritten(), I see that the next chunk of data are sent to socket. So, its behaviour is similar to QAbstractSocket::flush(). Cause I'm not using any QEventLoop, I'm using waitForBytesWritten()

@artwaw said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

@nico88desmo To add: if you don't need any sophisticated control and just want to send data, you can use QDataStream with a socket. All the data is buffered and send under to OS control.

Please be aware that the usual flush control is implemented into the socket: should you call disconnectFromHost() (the right way to close the connection) the buffer is sent out and only after then the connection is closed. This may introduce some waiting time.

So the idea is using QDataStream with QTcpSocket and wrap QByteArray / const char* directly to QDataStream, in order to not be worried about chunk handling and let's the OS take control of this, correct?

JonB

@nico88desmo
You can only use QDataStream if both sides of the connection are running a Qt program. Since you talk about "raw data to a device" I imagine that is not the case at device side, so you can cross off QDataStream.

I don't know why you want to count the bytes written/call waitForBytesWritten(), what issue are you trying to address? Normally you can just call a big write() and forget about it.

artwaw

@JonB I am sorry @JonB but what I meant by using QDataStream was relying on this: https://doc.qt.io/qt-6/qdatastream.html#reading-and-writing-raw-binary-data

Either I am unbelievably lucky in the past or indeed it doesn't require reader to understand QDataStream.

JonB

@artwaw
I have not verified this, so you need to do so yourself. I believe the "raw data" can still only be used within the context of a Qt QDataStream exchange. I say this because of QDataStream::setVersion(). I thought that this sends at least that version to the other end at the start of the exchange, which would mess up anything that wanted to send nothing but the data. Now I wonder whether that is not the case, and is only used internally at the sender/receiver side to identify its own version of QDataStream, and cannot be used to detect whether the other side is compatible (unless you manually send this version down the stream yourself).

I should be interested to know the actual situation if you feel like investigating yourself! I don't have a "stream sniffer" to look at the bytes sent myself.

UPDATE
I have now tested and agree that with QDataStream::writeRawData() and nothing else it does indeed send nothing but the raw bytes. No version header. Now that I know this I shall be more careful about saying "You can only use QDataStream if both sides of the connection are running a Qt program". Thank you for bringing this to my attention!

So I agree that would work OK here. However, in this case what advantage would wrapping inside a QDataStream offer over the existing simple socket.write() used by the OP? I don't see anything extra it would add? I believe it will take a copy of the 180MB, which I'm not sure achieves anything useful.

nico88desmo

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

@nico88desmo
You can only use QDataStream if both sides of the connection are running a Qt program. Since you talk about "raw data to a device" I imagine that is not the case at device side, so you can cross off QDataStream.

I don't know why you want to count the bytes written/call waitForBytesWritten(), what issue are you trying to address? Normally you can just call a big write() and forget about it.

The issue I have had is that, using a single big write, the data aren't not totally sent; instead, I see that only the first chunk of data is sent.
I solve this behaviour iterating over waitForBytesWritten() and checking, for each iteration, the remaing bytes to send using bytesToWrite().

I don't know if this appears because I'm using a synch-approach (so not using a QEventLoop)

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

@artwaw
I have not verified this, so you need to do so yourself. I believe the "raw data" can still only be used within the context of a Qt QDataStream exchange. I say this because of QDataStream::setVersion(). I thought that this sends at least that version to the other end at the start of the exchange, which would mess up anything that wanted to send nothing but the data. Now I wonder whether that is not the case, and is only used internally at the sender/receiver side to identify its own version of QDataStream, and cannot be used to detect whether the other side is compatible (unless you manually send this version down the stream yourself).

I should be interested to know the actual situation if you feel like investigating yourself! I don't have a "stream sniffer" to look at the bytes sent myself.

To the other side (an hardware device), I'm just having a bare buffer area, so surely I can't add any "tag/header" data.
I can try to use QDataStream in order to not use any kind of low level logic (chunks, residual bytes to send, ecc), but it's important that the stream sent is purely a raw stream, without any other info.

JonB

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

To the other side (an hardware device), I'm just having a bare buffer area, so surely I can't add any "tag/header" data.

I know this, that is why I am checking with @artwaw what it does per my reply to him.

The issue I have had is that, using a single big write, the data aren't not totally sent; instead, I see that only the first chunk of data is sent.

In what way "not totally sent"? You put in the comment:

        // **** bytesWritten here return 180 MB always ****
        bytesWritten = socket.write(stream + bytesWritten, streamSizeBytes - bytesWritten); // (1)

I take that at face value. It tells you it has sent the full 180MB. Why do you then impose a loop on calling waitForBytesWritten()/bytesToWrite()? Does your while (bytesToWrite > 0) loop ever return false;? So far as I can see you are blocking in sendData() until bytes are apparently written and I'm not sure why.

nico88desmo

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

To the other side (an hardware device), I'm just having a bare buffer area, so surely I can't add any "tag/header" data.

I know this, that is why I am checking with @artwaw what it does per my reply to him.

Ok, clear

The issue I have had is that, using a single big write, the data aren't not totally sent; instead, I see that only the first chunk of data is sent.

In what way "not totally sent"? You put in the comment:
        // **** bytesWritten here return 180 MB always ****
        bytesWritten = socket.write(stream + bytesWritten, streamSizeBytes - bytesWritten); // (1)
I take that at face value. It tells you it has sent the full 180MB. Why do you then impose a loop on calling waitForBytesWritten()/bytesToWrite()? Does your while (bytesToWrite > 0) loop ever return false;? So far as I can see you are blocking in sendData() until bytes are apparently written and I'm not sure why.

This is my doubt: socket.write() returns that I (theorically) sent 180 MB (that is why I write that comment), but this isn't true: in practice, it seems the value return from socket.write() doesn't represent the real data sent but, instead, the data that are "scheduled" to be sent.

At the beginning, I don't use any loop to check how many data have been sent (cause socket.write() returns 180 MB), but in practice, this isn't correct.

JonB

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

This is my doubt: socket.write() returns that I (theorically) sent 180 MB (that is why I write that comment), but this isn't true: in practice, it seems the value return from socket.write() doesn't represent the real data sent but, instead, the data that are "scheduled" to be sent.

Yes if you want to call it that it's what is "scheduled" to be sent, in that write() does not block until all the data is physically sent. Why should it? This is also physically true if, say, you write to a file instead of a socket. Actual commit to disk happens in the background over time. You only slow the OS down/block your calling program if you do, say, an fsync(). OSes buffer stuff for your benefit all over the place. The data is written whether you call waitForBytesWritten() or bytesToWrite() or not, so you have not explained why you wish to call them. If you want to that's OK, but I don't see the point much.

At the beginning, I don't use any loop to check how many data have been sent (cause socket.write() returns 180 MB), but in practice, this isn't correct.

What is not "correct"? You are only choosing to call it "not correct" because it does not tell you how many bytes have been physically sent at that point, so what/why do you care?

artwaw

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

what advantage would wrapping inside a QDataStream offer over the existing simple socket.write() used by the OP?

Programming convenience only, I think. But it is useful knowledge if one is more used to work with the streams. I am nitpicking here, my apologies.
To add (and finish this slight offtopic): version is sent/written only after the stream type is determined (raw/not raw) or otherwise explicitly set, I think. I came to this by using QDataStream in raw mode for binary things out of habit carried from other programming languages.

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

it seems the value return from socket.write() doesn't represent the real data sent but, instead, the data that are "scheduled" to be sent.

This is absolutely correct. Socket doesn't deal with hardware, it deals with OS. So, what it in fact reports, is "I've passed x amount of data to the underlying OS".

...and I see @JonB already explained this :)

It is somewhat similar to issues people come with when copying data on disk - very often actual write happens on closing, when write() reports all set and done.

nico88desmo

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

Yes if you want to call it that it's what is "scheduled" to be sent, in that write() does not block until all the data is physically sent. Why should it? This is also physically true if, say, you write to a file instead of a socket. Actual commit to disk happens in the background over time. You only slow the OS down/block your calling program if you do, say, an fsync(). OSes buffer stuff for your benefit all over the place. The data is written whether you call waitForBytesWritten() or bytesToWrite() or not, so you have not explained why you wish to call them. If you want to that's OK, but I don't see the point much.

Good explanation :) So, as you wrote, every buffer is commit in the background by OS policies; that's explain the behaviour I notice.
For my purpose, I'd like to flush that buffer before closing that method and before closing connection. That's why I call waitForBytesWritten() many times.

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

What is not "correct"? You are only choosing to call it "not correct" because it does not tell you how many bytes have been physically sent at that point, so what/why do you care?

Yes, I use the term "not correct" but, to be more precise, I should write "not expected as i thought".

@artwaw said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

To add (and finish this slight offtopic): version is sent/written only after the stream type is determined (raw/not raw) or otherwise explicitly set, I think. I came to this by using QDataStream in raw mode for binary things out of habit carried from other programming languages.

That's sounds perfect for my purpose; avoiding to use low-level logic is surely more robust. I'll try this way.

Thanks both @artwaw and @JonB for your responses.

JonB

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

For my purpose, I'd like to flush that buffer before closing that method and before closing connection

But why? :) It is always true that closing a connection (maybe not "aborting", but certainly closing) will flush any pending data. And I don't think any level of user code will actually flush the lowest level, physical transport of bytes down the socket and out of the machine, I believe at best it's really "written" out of user space and into OS space, where you still don't know when it will physically be sent.

Anyway, probably enough said now. You can do what you have in your code if you wish.

nico88desmo

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

But why? :)

Because, as I've written other code parts for this device communication, currently I don't close the connection after 180 MB of data has been sent to device (I have to send 180 MB of data per time for a continuous stream data, only waiting that this device has enough space available to receive it)

In any case, thanks for your replies; everything is clear now.

JonB

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

only waiting that this device has enough space available to receive it

I am not at all convinced that anything about write/waitForBytesWritten/bytesToWrite will tell you anything about whether the device at the other end "has enough space available to receive it". Have you tested that? Just a heads-up.

Christian Ehrlicher

@JonB It will for sure not. The data is somewhere between the users application and the other end. Maybe even buffered in a switch / router / whatsoever.