Still having trouble deserializing binary data from another UDP source (non-qt)
-
Here's the old topic: https://forum.qt.io/topic/135363/trouble-with-serializing-and-deserializing-structs
-
In the first screenshot, I'm concerned about the null characters at position 1,2,3 after the packet size. The packet size is supposed to be 8. What are those three null characters after the 8 before the first NAME value of TimeElapsed.
-
In the first screenshot, I'm concerned about the null characters at position 1,2,3 after the packet size. The packet size is supposed to be 8. What are those three null characters after the 8 before the first NAME value of TimeElapsed.
-
In the first screenshot, I'm concerned about the null characters at position 1,2,3 after the packet size. The packet size is supposed to be 8. What are those three null characters after the 8 before the first NAME value of TimeElapsed.
@RobbieP seems like typical Data Structure alignment issues.
-
HI all. I previously had some great help from @KroMignon on this topic, but anyone can feel free to jump in.
I'm writing a program to receive and log data from a data source in our lab. The data is structured as such:
#define SP_PACKET_SIZE 200 #define NAME_SIZE 64 struct fennecPacketStruct { int Size; ///< Number of items in the data packet char Name[SP_PACKET_SIZE][NAME_SIZE]; ///< Data labels double Value[SP_PACKET_SIZE]; ///< Data packet values };and it's sent via UDP in binary: (char *)fennecPacket
The sample code using QT to serialize and deserialize the packet works great. But it's having trouble deserializing the packet from the external source...so I'm wondering what changes I need to make to the deserializer to possibly fix it? Here's some screenshots showing the code and the Raw Packet data as well as the result of the initial deserialization.
void MainWindow::readPendingDatagrams() { fennecPacketStruct inPacket; while (udpSocket->hasPendingDatagrams()) { QHostAddress ThisHost; quint16 ThisPort; QByteArray ThisPacket; ThisPacket.resize(udpSocket->pendingDatagramSize()); udpSocket->readDatagram(ThisPacket.data(), ThisPacket.size(), &ThisHost, &ThisPort); QDataStream deserialize(ThisPacket); deserialize >> inPacket.Size; if(inPacket.Size > 0) { deserialize.readRawData(reinterpret_cast<char *>(inPacket.Name), NAME_SIZE * inPacket.Size); deserialize.readRawData(reinterpret_cast<char *>(inPacket.Value), sizeof(double) * inPacket.Size); }-
What exactly is going wrong with your code (if anything)?
-
Does your code work right when
inPacket.Size == 1? And only go wrong when larger than that? Or always? Or what? -
You thanked @KroMignon for his code in https://forum.qt.io/topic/135363/trouble-with-serializing-and-deserializing-structs/6, saying it was working for you. You seem to be using the same code. What has changed since then?
-
-
-
What exactly is going wrong with your code (if anything)?
-
Does your code work right when
inPacket.Size == 1? And only go wrong when larger than that? Or always? Or what? -
You thanked @KroMignon for his code in https://forum.qt.io/topic/135363/trouble-with-serializing-and-deserializing-structs/6, saying it was working for you. You seem to be using the same code. What has changed since then?
@JonB the code hasn't changed since that original post...what has changed is that in that sample code, I was serializing the data in qt and deserializing it in qt...same program actually, just sent over localhost to itself. This error has occurred while trying to process the data from another program, which is how it was initially supposed to work in the first place. The error I get is a seg fault on the second deserialize.readRawData
I'll try the packet size of 1 and see what happens.
-
-
-
What exactly is going wrong with your code (if anything)?
-
Does your code work right when
inPacket.Size == 1? And only go wrong when larger than that? Or always? Or what? -
You thanked @KroMignon for his code in https://forum.qt.io/topic/135363/trouble-with-serializing-and-deserializing-structs/6, saying it was working for you. You seem to be using the same code. What has changed since then?
-
-
@JonB do I need to change my deserialize to QDataStream::LittleEndian? It's currently BigEndian
@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):
do I need to change my deserialize to QDataStream::LittleEndian? It's currently BigEndian
How should we know this? Is your remote application using big or little endian byte order?
[EDIT]: you could do a kind of "sanity check" with:
deserialize >> inPacket.Size; qDebug() << "Readed size:" << inPacket.Size; int expectedSize = (ThisPacket.length() - sizeof(int))/(NAME_SIZE +sizeof(double)); qDebug() << "Expected size:" << expectedSize; -
@JonB do I need to change my deserialize to QDataStream::LittleEndian? It's currently BigEndian
@RobbieP
Hmm. In principle it does not matter so long as both sender & receiver are using same-endianness.Having said that, bytes you show of
8, 0, 0, 0look little-endian to me, from the sender's side??In any case
deserialize >> inPacket.Size; qDebug() << inPacket.Size;Put
qDebug()s in while you are developing deserialization! Either this will print8, which is good, or it will print millions & billions, in which caseNAME_SIZE * inPacket.Sizeetc. will be really bad (and cause seg faults)! :)Also int QDataStream::readRawData(char *s, int len) "returns the number of bytes read", which you should be printing out/checking....
-
@RobbieP
Hmm. In principle it does not matter so long as both sender & receiver are using same-endianness.Having said that, bytes you show of
8, 0, 0, 0look little-endian to me, from the sender's side??In any case
deserialize >> inPacket.Size; qDebug() << inPacket.Size;Put
qDebug()s in while you are developing deserialization! Either this will print8, which is good, or it will print millions & billions, in which caseNAME_SIZE * inPacket.Sizeetc. will be really bad (and cause seg faults)! :)Also int QDataStream::readRawData(char *s, int len) "returns the number of bytes read", which you should be printing out/checking....
-
@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):
@JonB the inPacket.Size is being read as 134217728
:) As suspected. My "millions & billions"! Wrong endianness, sender & receiver are reversed, they must agree....
134217728 == 0x08000000! Your little-endian-saved number is being read as big-endian.... -
@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):
@JonB the inPacket.Size is being read as 134217728
:) As suspected. My "millions & billions"! Wrong endianness, sender & receiver are reversed, they must agree....
134217728 == 0x08000000! Your little-endian-saved number is being read as big-endian.... -
@JonB the code hasn't changed since that original post...what has changed is that in that sample code, I was serializing the data in qt and deserializing it in qt...same program actually, just sent over localhost to itself. This error has occurred while trying to process the data from another program, which is how it was initially supposed to work in the first place. The error I get is a seg fault on the second deserialize.readRawData
I'll try the packet size of 1 and see what happens.
@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):
same program actually, just sent over localhost to itself. This error has occurred while trying to process the data from another program
When sending over localhost to self the endianness of the machine does not change, so defaults should be fine. Either this test was from another, different-architecture machine, or if it is same machine you must go and look at the sender's Qt code. Your receiver must be set to agree with that. It looks like "the data from another program" was explicitly set to little-endian, you will need that if it's not your default.
Meanwhile your receiver code should have a look at QDataStream::ByteOrder QDataStream::byteOrder() const
Returns the current byte order setting -- either BigEndian or LittleEndian.
Oh, I see void QDataStream::setByteOrder(QDataStream::ByteOrder bo)
The default setting is big endian. We recommend leaving this setting unless you have special requirements.
I think you said that's what your receiver end has. So it looks like the sender side explicitly changed that?
-
The Sender isn't written in QT...it's just using regular socket sending. I'm not sure it's explicitly setting the endianness but I'll look.
When I set
deserialize.setByteOrder(QDataStream::LittleEndian);I get a size of 8, which is correct. And I get all the name values correctly. But now it's not properly deserializing the doubles in the value section. So close i can taste it though. I really appreciate the help. I'm starting to wrap my mind around this stuff now, but doing it in QT is still pretty new to me.
-
The Sender isn't written in QT...it's just using regular socket sending. I'm not sure it's explicitly setting the endianness but I'll look.
When I set
deserialize.setByteOrder(QDataStream::LittleEndian);I get a size of 8, which is correct. And I get all the name values correctly. But now it's not properly deserializing the doubles in the value section. So close i can taste it though. I really appreciate the help. I'm starting to wrap my mind around this stuff now, but doing it in QT is still pretty new to me.
@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):
The Sender isn't written in QT...it's just using regular socket sending.
Whaaattt!!?? Why did you not say this from the outset? You cannot/must not/should not attempt to use
QDataStreamfor anything other thanQDataStream<->QDataStreamexchange. It puts in its own meta-data.Back to the drawing board. Remove
QDataStream. Do whatever coding you need for whatever the other end sends in whatever format it uses....If you insist on using
QDataStream, just using raw byte stuff, I don't much see the point but it's up to you. Certainly thendeserialize >> inPacket.Size;is pretty dodgy. -
Sorry I thought I mentioned it...I looked back up and it seems I just said "sent in UDP" I meant to say it's using a custom udp sender and the packet is just sent as (char*)packet without going through a serializer. So...a little background information. The program I'm replacing with a QT app was multicast...it was destroying our network, but I don't have the source code for it. Only the source code for the sender. I've modified the sender (which is a plugin for the X-Plane flight simulator, so I can't use QT) to only do unicast. And I've rewritten the receiver in QT. So to be honest...I've never deserialized without qt. Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?
-
Sorry I thought I mentioned it...I looked back up and it seems I just said "sent in UDP" I meant to say it's using a custom udp sender and the packet is just sent as (char*)packet without going through a serializer. So...a little background information. The program I'm replacing with a QT app was multicast...it was destroying our network, but I don't have the source code for it. Only the source code for the sender. I've modified the sender (which is a plugin for the X-Plane flight simulator, so I can't use QT) to only do unicast. And I've rewritten the receiver in QT. So to be honest...I've never deserialized without qt. Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?
@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):
Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?
Nope, because of possible packing/data alignment issues....
At the end of the day you must know, somehow, how the data was sent/formatted. You will also need to know if it sends things like
ints, and what endianness it does that in. As in: without that you can't read the data back sensibly.I don't know whether others would say you can still use
QDataStream::readRawData()safely, or if you did whether you are then getting much benefit out ofQDataStreamanyway.If it were me I would just write code to read chunks of bytes as per whatever the sender's format is. Then I know where I am for sure. Qt/C++/C have functions to transform between endian integers without having to use
QDataStream. Suggest do not read into astructdirectly, read into each member separately as per what you were doing. Then alignment at receiving size won't be an issue. But without looking at sending code's side, can't be sure whether that might have padding/alignment. If we assume sender itself did not e.g. send anystructs as a whole you may be good.Off you go, and do lots of testing on sender's packets!
I've never deserialized without qt
OK, if you really want to try sticking with it. Just tell your receiver
QDataStreamto be little-endian. See whether that sorts the current issue to your satisfaction. If so, try lots of others. If they all work, you can stick withQDataStreamI guess.... -
@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):
Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?
Nope, because of possible packing/data alignment issues....
At the end of the day you must know, somehow, how the data was sent/formatted. You will also need to know if it sends things like
ints, and what endianness it does that in. As in: without that you can't read the data back sensibly.I don't know whether others would say you can still use
QDataStream::readRawData()safely, or if you did whether you are then getting much benefit out ofQDataStreamanyway.If it were me I would just write code to read chunks of bytes as per whatever the sender's format is. Then I know where I am for sure. Qt/C++/C have functions to transform between endian integers without having to use
QDataStream. Suggest do not read into astructdirectly, read into each member separately as per what you were doing. Then alignment at receiving size won't be an issue. But without looking at sending code's side, can't be sure whether that might have padding/alignment. If we assume sender itself did not e.g. send anystructs as a whole you may be good.Off you go, and do lots of testing on sender's packets!
I've never deserialized without qt
OK, if you really want to try sticking with it. Just tell your receiver
QDataStreamto be little-endian. See whether that sorts the current issue to your satisfaction. If so, try lots of others. If they all work, you can stick withQDataStreamI guess.... -
@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):
Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?
Nope, because of possible packing/data alignment issues....
At the end of the day you must know, somehow, how the data was sent/formatted. You will also need to know if it sends things like
ints, and what endianness it does that in. As in: without that you can't read the data back sensibly.I don't know whether others would say you can still use
QDataStream::readRawData()safely, or if you did whether you are then getting much benefit out ofQDataStreamanyway.If it were me I would just write code to read chunks of bytes as per whatever the sender's format is. Then I know where I am for sure. Qt/C++/C have functions to transform between endian integers without having to use
QDataStream. Suggest do not read into astructdirectly, read into each member separately as per what you were doing. Then alignment at receiving size won't be an issue. But without looking at sending code's side, can't be sure whether that might have padding/alignment. If we assume sender itself did not e.g. send anystructs as a whole you may be good.Off you go, and do lots of testing on sender's packets!
I've never deserialized without qt
OK, if you really want to try sticking with it. Just tell your receiver
QDataStreamto be little-endian. See whether that sorts the current issue to your satisfaction. If so, try lots of others. If they all work, you can stick withQDataStreamI guess....
