[QTcpSocket - Qt-6.2.7] Write to socket big raw data

[JonB]

@nico88desmo
You can only use QDataStream if both sides of the connection are running a Qt program. Since you talk about "raw data to a device" I imagine that is not the case at device side, so you can cross off QDataStream.

I don't know why you want to count the bytes written/call waitForBytesWritten(), what issue are you trying to address? Normally you can just call a big write() and forget about it.

[artwaw]

@JonB I am sorry @JonB but what I meant by using QDataStream was relying on this: https://doc.qt.io/qt-6/qdatastream.html#reading-and-writing-raw-binary-data

Either I am unbelievably lucky in the past or indeed it doesn't require reader to understand QDataStream.

[JonB]

@artwaw
I have not verified this, so you need to do so yourself. I believe the "raw data" can still only be used within the context of a Qt QDataStream exchange. I say this because of QDataStream::setVersion(). I thought that this sends at least that version to the other end at the start of the exchange, which would mess up anything that wanted to send nothing but the data. Now I wonder whether that is not the case, and is only used internally at the sender/receiver side to identify its own version of QDataStream, and cannot be used to detect whether the other side is compatible (unless you manually send this version down the stream yourself).

I should be interested to know the actual situation if you feel like investigating yourself! I don't have a "stream sniffer" to look at the bytes sent myself.

UPDATE
I have now tested and agree that with QDataStream::writeRawData() and nothing else it does indeed send nothing but the raw bytes. No version header. Now that I know this I shall be more careful about saying "You can only use QDataStream if both sides of the connection are running a Qt program". Thank you for bringing this to my attention!

So I agree that would work OK here. However, in this case what advantage would wrapping inside a QDataStream offer over the existing simple socket.write() used by the OP? I don't see anything extra it would add? I believe it will take a copy of the 180MB, which I'm not sure achieves anything useful.

[nico88desmo]

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

@nico88desmo
You can only use QDataStream if both sides of the connection are running a Qt program. Since you talk about "raw data to a device" I imagine that is not the case at device side, so you can cross off QDataStream.

I don't know why you want to count the bytes written/call waitForBytesWritten(), what issue are you trying to address? Normally you can just call a big write() and forget about it.

The issue I have had is that, using a single big write, the data aren't not totally sent; instead, I see that only the first chunk of data is sent.
I solve this behaviour iterating over waitForBytesWritten() and checking, for each iteration, the remaing bytes to send using bytesToWrite().

I don't know if this appears because I'm using a synch-approach (so not using a QEventLoop)

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

@artwaw
I have not verified this, so you need to do so yourself. I believe the "raw data" can still only be used within the context of a Qt QDataStream exchange. I say this because of QDataStream::setVersion(). I thought that this sends at least that version to the other end at the start of the exchange, which would mess up anything that wanted to send nothing but the data. Now I wonder whether that is not the case, and is only used internally at the sender/receiver side to identify its own version of QDataStream, and cannot be used to detect whether the other side is compatible (unless you manually send this version down the stream yourself).

I should be interested to know the actual situation if you feel like investigating yourself! I don't have a "stream sniffer" to look at the bytes sent myself.

To the other side (an hardware device), I'm just having a bare buffer area, so surely I can't add any "tag/header" data.
I can try to use QDataStream in order to not use any kind of low level logic (chunks, residual bytes to send, ecc), but it's important that the stream sent is purely a raw stream, without any other info.

[JonB]

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

To the other side (an hardware device), I'm just having a bare buffer area, so surely I can't add any "tag/header" data.

I know this, that is why I am checking with @artwaw what it does per my reply to him.

The issue I have had is that, using a single big write, the data aren't not totally sent; instead, I see that only the first chunk of data is sent.

In what way "not totally sent"? You put in the comment:

        // **** bytesWritten here return 180 MB always ****
        bytesWritten = socket.write(stream + bytesWritten, streamSizeBytes - bytesWritten); // (1)

I take that at face value. It tells you it has sent the full 180MB. Why do you then impose a loop on calling waitForBytesWritten()/bytesToWrite()? Does your while (bytesToWrite > 0) loop ever return false;? So far as I can see you are blocking in sendData() until bytes are apparently written and I'm not sure why.

[nico88desmo]

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

To the other side (an hardware device), I'm just having a bare buffer area, so surely I can't add any "tag/header" data.

I know this, that is why I am checking with @artwaw what it does per my reply to him.

Ok, clear

The issue I have had is that, using a single big write, the data aren't not totally sent; instead, I see that only the first chunk of data is sent.

In what way "not totally sent"? You put in the comment:

        // **** bytesWritten here return 180 MB always ****
        bytesWritten = socket.write(stream + bytesWritten, streamSizeBytes - bytesWritten); // (1)

I take that at face value. It tells you it has sent the full 180MB. Why do you then impose a loop on calling waitForBytesWritten()/bytesToWrite()? Does your while (bytesToWrite > 0) loop ever return false;? So far as I can see you are blocking in sendData() until bytes are apparently written and I'm not sure why.

This is my doubt: socket.write() returns that I (theorically) sent 180 MB (that is why I write that comment), but this isn't true: in practice, it seems the value return from socket.write() doesn't represent the real data sent but, instead, the data that are "scheduled" to be sent.

At the beginning, I don't use any loop to check how many data have been sent (cause socket.write() returns 180 MB), but in practice, this isn't correct.

[JonB]

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

This is my doubt: socket.write() returns that I (theorically) sent 180 MB (that is why I write that comment), but this isn't true: in practice, it seems the value return from socket.write() doesn't represent the real data sent but, instead, the data that are "scheduled" to be sent.

Yes if you want to call it that it's what is "scheduled" to be sent, in that write() does not block until all the data is physically sent. Why should it? This is also physically true if, say, you write to a file instead of a socket. Actual commit to disk happens in the background over time. You only slow the OS down/block your calling program if you do, say, an fsync(). OSes buffer stuff for your benefit all over the place. The data is written whether you call waitForBytesWritten() or bytesToWrite() or not, so you have not explained why you wish to call them. If you want to that's OK, but I don't see the point much.

At the beginning, I don't use any loop to check how many data have been sent (cause socket.write() returns 180 MB), but in practice, this isn't correct.

What is not "correct"? You are only choosing to call it "not correct" because it does not tell you how many bytes have been physically sent at that point, so what/why do you care?

[artwaw]

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

what advantage would wrapping inside a QDataStream offer over the existing simple socket.write() used by the OP?

Programming convenience only, I think. But it is useful knowledge if one is more used to work with the streams. I am nitpicking here, my apologies.
To add (and finish this slight offtopic): version is sent/written only after the stream type is determined (raw/not raw) or otherwise explicitly set, I think. I came to this by using QDataStream in raw mode for binary things out of habit carried from other programming languages.

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

it seems the value return from socket.write() doesn't represent the real data sent but, instead, the data that are "scheduled" to be sent.

This is absolutely correct. Socket doesn't deal with hardware, it deals with OS. So, what it in fact reports, is "I've passed x amount of data to the underlying OS".

...and I see @JonB already explained this :)

It is somewhat similar to issues people come with when copying data on disk - very often actual write happens on closing, when write() reports all set and done.

[nico88desmo]

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

Yes if you want to call it that it's what is "scheduled" to be sent, in that write() does not block until all the data is physically sent. Why should it? This is also physically true if, say, you write to a file instead of a socket. Actual commit to disk happens in the background over time. You only slow the OS down/block your calling program if you do, say, an fsync(). OSes buffer stuff for your benefit all over the place. The data is written whether you call waitForBytesWritten() or bytesToWrite() or not, so you have not explained why you wish to call them. If you want to that's OK, but I don't see the point much.

Good explanation :) So, as you wrote, every buffer is commit in the background by OS policies; that's explain the behaviour I notice.
For my purpose, I'd like to flush that buffer before closing that method and before closing connection. That's why I call waitForBytesWritten() many times.

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

What is not "correct"? You are only choosing to call it "not correct" because it does not tell you how many bytes have been physically sent at that point, so what/why do you care?

Yes, I use the term "not correct" but, to be more precise, I should write "not expected as i thought".

@artwaw said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

To add (and finish this slight offtopic): version is sent/written only after the stream type is determined (raw/not raw) or otherwise explicitly set, I think. I came to this by using QDataStream in raw mode for binary things out of habit carried from other programming languages.

That's sounds perfect for my purpose; avoiding to use low-level logic is surely more robust. I'll try this way.

Thanks both @artwaw and @JonB for your responses.

[JonB]

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

For my purpose, I'd like to flush that buffer before closing that method and before closing connection

But why? :) It is always true that closing a connection (maybe not "aborting", but certainly closing) will flush any pending data. And I don't think any level of user code will actually flush the lowest level, physical transport of bytes down the socket and out of the machine, I believe at best it's really "written" out of user space and into OS space, where you still don't know when it will physically be sent.

Anyway, probably enough said now. You can do what you have in your code if you wish.

[nico88desmo]

@JonB said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

But why? :)

Because, as I've written other code parts for this device communication, currently I don't close the connection after 180 MB of data has been sent to device (I have to send 180 MB of data per time for a continuous stream data, only waiting that this device has enough space available to receive it)

In any case, thanks for your replies; everything is clear now.

[JonB]

@nico88desmo said in [QTcpSocket - Qt-6.2.7] Write to socket big raw data:

only waiting that this device has enough space available to receive it

I am not at all convinced that anything about write/waitForBytesWritten/bytesToWrite will tell you anything about whether the device at the other end "has enough space available to receive it". Have you tested that? Just a heads-up.

[Christian Ehrlicher]

@JonB It will for sure not. The data is somewhere between the users application and the other end. Maybe even buffered in a switch / router / whatsoever.