reinterpret_cast for received tcp packets

Redman

@J-Hilk Is there anything you can recommend for me to read to get a better understanding regarding this topic?
While I understand your statement, this still leaves open my question regarding any incompatibilities between python and c++.

J.Hilk

@Redman I'm afraid most of my experience is from personally falling into such pits myself :D

But for this type punning case I think this video might be a good source, its part of my link collection:
https://youtu.be/_qzMpk-22cc

Christian Ehrlicher

Fix your python code so it sends the struct with the correct size for the members. Or adjust your code to read uint32 for every part. Normally there is a spec where you define the format on the wire.

JonB

@Redman said in reinterpret_cast for received tcp packets:

No need for checking if the tcpSocker buffer is empty or if the available bytes in the buffer are only partially tansmitted etc.etc. Simply readAll bytes and cast it to the desired struct.

Are you doing this as the communication runs along during the program, or only at the end after you have closed the socket? Assuming it's as you go along, how do you know that readAll() returns all bytes which might have been sent? It only guarantees to return 1 byte when readyRead() signal is fired, you have no guarantee any further bytes are there. The fact that it seems to work in some situation does not mean it won't go wrong in another. This has nothing to do with the casting.

What do you use in Python to pack the data and send it, and how that compares to your struct's size/order/padding? What endian-ness is the sender machine and order of bytes sent?

GrecKo

or just use gRPC or an alternative.

Redman

The problem seems to be the byte-alignment for the struct. The previously used c++ app that sent the packets used MSVC aswell as I do.

#pragma pack(push, 1)
struct Coordinate {
   quint32 id;
   quint16 mode;
   qint32 x;
   quint64 y;
   qint32 z;
};
#pragma pack(pop)

Fixed the problem.

J.Hilk

calling that fixed is generous :P

But if you're satisfied! 👍

Redman

@JonB I do this during the whole execution time of my application. The tcp packet size is only 22 bytes, so fairly small.
How would I go about making sure I get all the bytes out of the buffer?

Redman

@J-Hilk As far as I understood it either its me who has to adjust or the sender. Why would you consider this to be a generous definition of fixed?

jsulm

@Redman said in reinterpret_cast for received tcp packets:

Why would you consider this to be a generous definition of fixed?

What about indianess?

Redman

@jsulm Since both participants are Windows, little-endian is to be assumed. We did in fact switch to big-endian for each part of the message. This resulted in the first 2 parts not being recognized as a 1 anymore.

The fact, that in the 3rd part a 1 was sent but was cast to a 65536 indicates the byte-alignment problem.
Now Im not sure why @J-Hilk said it was a generous definition of fixed.

J.Hilk

@Redman said in reinterpret_cast for received tcp packets:

Why would you consider this to be a generous definition of fixed?

what about different compiler? pragma is highly depending on compiler and compiler version

also casting a char* to a struct is still UB and just a dud waiting to explode

Redman

@J-Hilk Since there are no plans of switching to a different Compiler than MSVC I did not consider that.
I would assume that

#pragma pack(push, 1)
#pragma pack(pop)

would do the same through all MSVC versions.

Regarding that reinterpret_cast, I will, as you suggested, implement a ctor which takes a QByteArray.
In fact, I already tried that out during figuring out what the actual problem was. To me the ctor looked "ugly".
Doing byteArray.sliced(i,n) to retrieve the corresponding bytes and casting them to the desired type might be a quick thing to implement for such a small struct. But for larger structs this gets ugly very fast.

Since c++ doesn't offer reflection, can I somehow use Qt's meta-object system to achieve a dynamic construction of my object?

jsulm

I also would not assume indianess based on operating system. Windows also runs on ARM and ARM can be both big or little.

Redman

@jsulm Thats why we tested both :)

J.Hilk

@Redman I would suggest using QDataStream

Christian Ehrlicher

Casting a struct to a received bunch of bytes is not correct. It may work now but fail as soon as a small piece changes. Therefore you have to describe the bytes on the paket (e.g. in an interface document) and decode them accordingly. Everything else is a hack.

JonB

@Redman said in reinterpret_cast for received tcp packets:

@JonB I do this during the whole execution time of my application. The tcp packet size is only 22 bytes, so fairly small.
How would I go about making sure I get all the bytes out of the buffer?

Assuming recEchoDataReady() is called as a slot from readyRead() signal, what you are supposed to do is append the bytes to a "pending" buffer and then check whether the total number read into pending equals or exceeds the number you need/expect for your struct. If so remove those from pending. Not only might you be "short" (read only 1+ bytes when 22 expected), you might also "overflow" (read e.g. 32 when 22 wanted for one packet because it also finds some of the next packet), in which case that packet will "work" but the next packet will go wrong.

If you don't want to do this, because you believe it is always 22, you might at least put in a QASSERT(bytes_read == sizeof(Coordinate)) to verify that never goes wrong.

You say a lot about the struct size/packing/padding/endianness etc. at the receiving C++ side but did not answer how you send the data from the Python side. If I understand right you have looked at "the wire" or the bytes you receive and say they don't look right as raw bytes? In which case check the sender side. I believe you are saying that it used to work when the sender was C++ but now goes wrong when the sender is Python, in which case the receiver side is irrelevant and the bytes sent must be different, if that is really the case.

01:00:00:00:01:00:01:00:00:00:01:00:00:00:00:00:00:00:01:00:00:00

Unfortunately after casting that ByteArray to my struct I get following values

1 1 65536 65536 -1870011232

While this seems to contradict the statement that it is the sender side which has changed/gone wrong, I would examine this more closely. Off the top of my head that data buffer would not produce the (packed) struct values you say it does.

You also wrote at one point:

@Redman said in reinterpret_cast for received tcp packets:

Fixed the problem.

Does this mean you no longer have any problem sending/receiving the structure? You introduced the #pragma pack(push, 1), did that fix, was that at C++ receiver side or what did you do at Python sender side? Your C struct needs to be packed if you expect it to be 22 bytes size.

Redman

@Christian-Ehrlicher The interface is well documented and both sites agreed to it. So there will be no change to that without further discussion between me and our partner :)

J.Hilk

@Redman said in reinterpret_cast for received tcp packets:

The interface is well documented and both sites agreed to it. So there will be no change to that

alt text