Still having trouble deserializing binary data from another UDP source (non-qt)

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

What are those three null characters after the 8 before the first NAME value of TimeElapsed.

4-byte/32-bit int, LSB-first, little-endian.

[J.Hilk]

@RobbieP seems like typical Data Structure alignment issues.

[JonB]

@RobbieP

• What exactly is going wrong with your code (if anything)?

• Does your code work right when inPacket.Size == 1? And only go wrong when larger than that? Or always? Or what?

• You thanked @KroMignon for his code in https://forum.qt.io/topic/135363/trouble-with-serializing-and-deserializing-structs/6, saying it was working for you. You seem to be using the same code. What has changed since then?

[RobbieP]

@JonB the code hasn't changed since that original post...what has changed is that in that sample code, I was serializing the data in qt and deserializing it in qt...same program actually, just sent over localhost to itself. This error has occurred while trying to process the data from another program, which is how it was initially supposed to work in the first place. The error I get is a seg fault on the second deserialize.readRawData

I'll try the packet size of 1 and see what happens.

[RobbieP]

@JonB do I need to change my deserialize to QDataStream::LittleEndian? It's currently BigEndian

[KroMignon]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

do I need to change my deserialize to QDataStream::LittleEndian? It's currently BigEndian

How should we know this? Is your remote application using big or little endian byte order?

[EDIT]: you could do a kind of "sanity check" with:

deserialize >> inPacket.Size;
qDebug() << "Readed size:" <<  inPacket.Size;
int expectedSize = (ThisPacket.length()  - sizeof(int))/(NAME_SIZE +sizeof(double));
qDebug() << "Expected size:" <<  expectedSize;

[JonB]

@RobbieP
Hmm. In principle it does not matter so long as both sender & receiver are using same-endianness.

Having said that, bytes you show of 8, 0, 0, 0 look little-endian to me, from the sender's side??

In any case

deserialize >> inPacket.Size;
qDebug() << inPacket.Size;

Put qDebug()s in while you are developing deserialization! Either this will print 8, which is good, or it will print millions & billions, in which case NAME_SIZE * inPacket.Size etc. will be really bad (and cause seg faults)! :)

Also int QDataStream::readRawData(char *s, int len) "returns the number of bytes read", which you should be printing out/checking....

[RobbieP]

@JonB the inPacket.Size is being read as 134217728

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

@JonB the inPacket.Size is being read as 134217728

:) As suspected. My "millions & billions"! Wrong endianness, sender & receiver are reversed, they must agree....

134217728 == 0x08000000! Your little-endian-saved number is being read as big-endian....

[RobbieP]

@JonB oh man...how do I fix that? Do I just set the endianness of the QDataStream?

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

same program actually, just sent over localhost to itself. This error has occurred while trying to process the data from another program

When sending over localhost to self the endianness of the machine does not change, so defaults should be fine. Either this test was from another, different-architecture machine, or if it is same machine you must go and look at the sender's Qt code. Your receiver must be set to agree with that. It looks like "the data from another program" was explicitly set to little-endian, you will need that if it's not your default.

Meanwhile your receiver code should have a look at QDataStream::ByteOrder QDataStream::byteOrder() const

Returns the current byte order setting -- either BigEndian or LittleEndian.

Oh, I see void QDataStream::setByteOrder(QDataStream::ByteOrder bo)

The default setting is big endian. We recommend leaving this setting unless you have special requirements.

I think you said that's what your receiver end has. So it looks like the sender side explicitly changed that?

[RobbieP]

The Sender isn't written in QT...it's just using regular socket sending. I'm not sure it's explicitly setting the endianness but I'll look.

When I set

     deserialize.setByteOrder(QDataStream::LittleEndian);

I get a size of 8, which is correct. And I get all the name values correctly. But now it's not properly deserializing the doubles in the value section. So close i can taste it though. I really appreciate the help. I'm starting to wrap my mind around this stuff now, but doing it in QT is still pretty new to me.

[RobbieP]

The doubles are all coming in as 0 or very close to 0.

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

The Sender isn't written in QT...it's just using regular socket sending.

Whaaattt!!?? Why did you not say this from the outset? You cannot/must not/should not attempt to use QDataStream for anything other than QDataStream<->QDataStream exchange. It puts in its own meta-data.

Back to the drawing board. Remove QDataStream. Do whatever coding you need for whatever the other end sends in whatever format it uses....

If you insist on using QDataStream, just using raw byte stuff, I don't much see the point but it's up to you. Certainly then deserialize >> inPacket.Size; is pretty dodgy.

[RobbieP]

Sorry I thought I mentioned it...I looked back up and it seems I just said "sent in UDP" I meant to say it's using a custom udp sender and the packet is just sent as (char*)packet without going through a serializer. So...a little background information. The program I'm replacing with a QT app was multicast...it was destroying our network, but I don't have the source code for it. Only the source code for the sender. I've modified the sender (which is a plugin for the X-Plane flight simulator, so I can't use QT) to only do unicast. And I've rewritten the receiver in QT. So to be honest...I've never deserialized without qt. Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?

Nope, because of possible packing/data alignment issues....

At the end of the day you must know, somehow, how the data was sent/formatted. You will also need to know if it sends things like ints, and what endianness it does that in. As in: without that you can't read the data back sensibly.

I don't know whether others would say you can still use QDataStream::readRawData() safely, or if you did whether you are then getting much benefit out of QDataStream anyway.

If it were me I would just write code to read chunks of bytes as per whatever the sender's format is. Then I know where I am for sure. Qt/C++/C have functions to transform between endian integers without having to use QDataStream. Suggest do not read into a struct directly, read into each member separately as per what you were doing. Then alignment at receiving size won't be an issue. But without looking at sending code's side, can't be sure whether that might have padding/alignment. If we assume sender itself did not e.g. send any structs as a whole you may be good.

Off you go, and do lots of testing on sender's packets!

I've never deserialized without qt

OK, if you really want to try sticking with it. Just tell your receiver QDataStream to be little-endian. See whether that sorts the current issue to your satisfaction. If so, try lots of others. If they all work, you can stick with QDataStream I guess....

[RobbieP]

@JonB I'm getting the correct names and the size...the doubles are off though...does the second readRawData start where the previous readRawData left off? calling readRawData back to back like that doesn't reset the position does it?

[RobbieP]

@JonB I ask because I'm curious if it's possible the second readRawData isn't tryign to convert the 200 64 character names into doubles.

[JonB]

@RobbieP
No, your readRawData()s run sequentially, carrying on from where they got to.

Possibilities:

• Sender does have some pad between end of Name and start of Value.
• Sender saves doubles as something other than write(&double, sizeof(double)).
• Sender is different architecture which represents doubles differently. Though I kind of doubt this, I believe there is some standard for how doubles are represented which is cross-architecture.
• QDataStream encodes double with some extra information. Though I doubt this.

the doubles are off though

How "off" is "off"? Slightly wrong, or way out there?

Suggestions:

• You really should put those checks in on return result of QDataStream::readRawData().
• Can you get the sender to send just with inPacket.Size = 1 while you get it sorted out?
• Try it with plain read()s and no QDataStream. Though if that does not work you're a bit stuck.
• Do you have the source code of the sender's write()ing code to examine?

[RobbieP]

@JonB

/* This method writes data to the UDP socket and returns the number of
* bytes written or -1 for an error.  Pass in the IP and port youare sending to. */

int PCSBSocketUDP::WriteData(
				void *				inBuf,
				int				inBufLength,
				unsigned int		inDstIP,
				unsigned short		inDstPort)
{
	sockaddr_in destAddress;
	int tolen = sizeof(destAddress);
	destAddress.sin_family = AF_INET;
	destAddress.sin_port = htons(inDstPort);
	destAddress.sin_addr.s_addr = htonl(inDstIP);
	int result = sendto(mWinSocket,
						 (char*)inBuf,
						 inBufLength,
						 0,
						 (sockaddr*)&destAddress,
						 tolen);

	if (result == -1)
	{
		return -1;
	}
	else
	{
		return result;
	}
}

and I'm sending the struct from the top of my first post with that using:

m_SendSocket->WriteData((char *)&m_Packet, sizeof (fennecPacketStruct), ntohl(inet_addr(m_ipaddress.c_str())), 46456);

the sender is running on this same linux machine, but it's a plugin to a flight simulator. I'll see what the return value is for readRawData next.