Still having trouble deserializing binary data from another UDP source (non-qt)

RobbieP

HI all. I previously had some great help from @KroMignon on this topic, but anyone can feel free to jump in.

I'm writing a program to receive and log data from a data source in our lab. The data is structured as such:

#define SP_PACKET_SIZE 200
#define NAME_SIZE 64
    struct fennecPacketStruct
    {
        int Size;                               ///< Number of items in the data packet
        char Name[SP_PACKET_SIZE][NAME_SIZE];   ///< Data labels
        double Value[SP_PACKET_SIZE];           ///< Data packet values
    };

and it's sent via UDP in binary: (char *)fennecPacket

The sample code using QT to serialize and deserialize the packet works great. But it's having trouble deserializing the packet from the external source...so I'm wondering what changes I need to make to the deserializer to possibly fix it? Here's some screenshots showing the code and the Raw Packet data as well as the result of the initial deserialization.

![0_1652188242645_Screenshot_2022-05-10_09-01-48_1920x1080.jpg](Uploading 100%)

void MainWindow::readPendingDatagrams()
{
    fennecPacketStruct inPacket;
    while (udpSocket->hasPendingDatagrams()) {
        QHostAddress ThisHost;
        quint16 ThisPort;
        QByteArray ThisPacket;
        ThisPacket.resize(udpSocket->pendingDatagramSize());
        udpSocket->readDatagram(ThisPacket.data(), ThisPacket.size(), &ThisHost, &ThisPort);
        QDataStream deserialize(ThisPacket);

        deserialize >> inPacket.Size;
        if(inPacket.Size > 0)
        {
            deserialize.readRawData(reinterpret_cast<char *>(inPacket.Name), NAME_SIZE * inPacket.Size);
            deserialize.readRawData(reinterpret_cast<char *>(inPacket.Value), sizeof(double) * inPacket.Size);
        }

RobbieP

Here's the old topic: https://forum.qt.io/topic/135363/trouble-with-serializing-and-deserializing-structs

RobbieP

In the first screenshot, I'm concerned about the null characters at position 1,2,3 after the packet size. The packet size is supposed to be 8. What are those three null characters after the 8 before the first NAME value of TimeElapsed.

JonB

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

What are those three null characters after the 8 before the first NAME value of TimeElapsed.

4-byte/32-bit int, LSB-first, little-endian.

J.Hilk

@RobbieP seems like typical Data Structure alignment issues.

JonB

@RobbieP

What exactly is going wrong with your code (if anything)?
Does your code work right when inPacket.Size == 1? And only go wrong when larger than that? Or always? Or what?
You thanked @KroMignon for his code in https://forum.qt.io/topic/135363/trouble-with-serializing-and-deserializing-structs/6, saying it was working for you. You seem to be using the same code. What has changed since then?

RobbieP

@JonB the code hasn't changed since that original post...what has changed is that in that sample code, I was serializing the data in qt and deserializing it in qt...same program actually, just sent over localhost to itself. This error has occurred while trying to process the data from another program, which is how it was initially supposed to work in the first place. The error I get is a seg fault on the second deserialize.readRawData

I'll try the packet size of 1 and see what happens.

RobbieP

@JonB do I need to change my deserialize to QDataStream::LittleEndian? It's currently BigEndian

KroMignon

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

do I need to change my deserialize to QDataStream::LittleEndian? It's currently BigEndian

How should we know this? Is your remote application using big or little endian byte order?

[EDIT]: you could do a kind of "sanity check" with:

deserialize >> inPacket.Size;
qDebug() << "Readed size:" <<  inPacket.Size;
int expectedSize = (ThisPacket.length()  - sizeof(int))/(NAME_SIZE +sizeof(double));
qDebug() << "Expected size:" <<  expectedSize;

JonB

@RobbieP
Hmm. In principle it does not matter so long as both sender & receiver are using same-endianness.

Having said that, bytes you show of 8, 0, 0, 0 look little-endian to me, from the sender's side??

In any case

deserialize >> inPacket.Size;
qDebug() << inPacket.Size;

Put qDebug()s in while you are developing deserialization! Either this will print 8, which is good, or it will print millions & billions, in which case NAME_SIZE * inPacket.Size etc. will be really bad (and cause seg faults)! :)

Also int QDataStream::readRawData(char *s, int len) "returns the number of bytes read", which you should be printing out/checking....

RobbieP

@JonB the inPacket.Size is being read as 134217728

JonB

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

@JonB the inPacket.Size is being read as 134217728

:) As suspected. My "millions & billions"! Wrong endianness, sender & receiver are reversed, they must agree....

134217728 == 0x08000000! Your little-endian-saved number is being read as big-endian....

RobbieP

@JonB oh man...how do I fix that? Do I just set the endianness of the QDataStream?

JonB

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

same program actually, just sent over localhost to itself. This error has occurred while trying to process the data from another program

When sending over localhost to self the endianness of the machine does not change, so defaults should be fine. Either this test was from another, different-architecture machine, or if it is same machine you must go and look at the sender's Qt code. Your receiver must be set to agree with that. It looks like "the data from another program" was explicitly set to little-endian, you will need that if it's not your default.

Meanwhile your receiver code should have a look at QDataStream::ByteOrder QDataStream::byteOrder() const

Returns the current byte order setting -- either BigEndian or LittleEndian.

Oh, I see void QDataStream::setByteOrder(QDataStream::ByteOrder bo)

The default setting is big endian. We recommend leaving this setting unless you have special requirements.

I think you said that's what your receiver end has. So it looks like the sender side explicitly changed that?

RobbieP

The Sender isn't written in QT...it's just using regular socket sending. I'm not sure it's explicitly setting the endianness but I'll look.

When I set

     deserialize.setByteOrder(QDataStream::LittleEndian);

I get a size of 8, which is correct. And I get all the name values correctly. But now it's not properly deserializing the doubles in the value section. So close i can taste it though. I really appreciate the help. I'm starting to wrap my mind around this stuff now, but doing it in QT is still pretty new to me.

RobbieP

The doubles are all coming in as 0 or very close to 0.

JonB

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

The Sender isn't written in QT...it's just using regular socket sending.

Whaaattt!!?? Why did you not say this from the outset? You cannot/must not/should not attempt to use QDataStream for anything other than QDataStream<->QDataStream exchange. It puts in its own meta-data.

Back to the drawing board. Remove QDataStream. Do whatever coding you need for whatever the other end sends in whatever format it uses....

If you insist on using QDataStream, just using raw byte stuff, I don't much see the point but it's up to you. Certainly then deserialize >> inPacket.Size; is pretty dodgy.

RobbieP

Sorry I thought I mentioned it...I looked back up and it seems I just said "sent in UDP" I meant to say it's using a custom udp sender and the packet is just sent as (char*)packet without going through a serializer. So...a little background information. The program I'm replacing with a QT app was multicast...it was destroying our network, but I don't have the source code for it. Only the source code for the sender. I've modified the sender (which is a plugin for the X-Plane flight simulator, so I can't use QT) to only do unicast. And I've rewritten the receiver in QT. So to be honest...I've never deserialized without qt. Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?

JonB

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?

Nope, because of possible packing/data alignment issues....

At the end of the day you must know, somehow, how the data was sent/formatted. You will also need to know if it sends things like ints, and what endianness it does that in. As in: without that you can't read the data back sensibly.

I don't know whether others would say you can still use QDataStream::readRawData() safely, or if you did whether you are then getting much benefit out of QDataStream anyway.

If it were me I would just write code to read chunks of bytes as per whatever the sender's format is. Then I know where I am for sure. Qt/C++/C have functions to transform between endian integers without having to use QDataStream. Suggest do not read into a struct directly, read into each member separately as per what you were doing. Then alignment at receiving size won't be an issue. But without looking at sending code's side, can't be sure whether that might have padding/alignment. If we assume sender itself did not e.g. send any structs as a whole you may be good.

Off you go, and do lots of testing on sender's packets!

I've never deserialized without qt

OK, if you really want to try sticking with it. Just tell your receiver QDataStream to be little-endian. See whether that sorts the current issue to your satisfaction. If so, try lots of others. If they all work, you can stick with QDataStream I guess....

RobbieP

@JonB I'm getting the correct names and the size...the doubles are off though...does the second readRawData start where the previous readRawData left off? calling readRawData back to back like that doesn't reset the position does it?