Still having trouble deserializing binary data from another UDP source (non-qt)

[RobbieP]

@JonB the inPacket.Size is being read as 134217728

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

@JonB the inPacket.Size is being read as 134217728

:) As suspected. My "millions & billions"! Wrong endianness, sender & receiver are reversed, they must agree....

134217728 == 0x08000000! Your little-endian-saved number is being read as big-endian....

[RobbieP]

@JonB oh man...how do I fix that? Do I just set the endianness of the QDataStream?

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

same program actually, just sent over localhost to itself. This error has occurred while trying to process the data from another program

When sending over localhost to self the endianness of the machine does not change, so defaults should be fine. Either this test was from another, different-architecture machine, or if it is same machine you must go and look at the sender's Qt code. Your receiver must be set to agree with that. It looks like "the data from another program" was explicitly set to little-endian, you will need that if it's not your default.

Meanwhile your receiver code should have a look at QDataStream::ByteOrder QDataStream::byteOrder() const

Returns the current byte order setting -- either BigEndian or LittleEndian.

Oh, I see void QDataStream::setByteOrder(QDataStream::ByteOrder bo)

The default setting is big endian. We recommend leaving this setting unless you have special requirements.

I think you said that's what your receiver end has. So it looks like the sender side explicitly changed that?

[RobbieP]

The Sender isn't written in QT...it's just using regular socket sending. I'm not sure it's explicitly setting the endianness but I'll look.

When I set

     deserialize.setByteOrder(QDataStream::LittleEndian);

I get a size of 8, which is correct. And I get all the name values correctly. But now it's not properly deserializing the doubles in the value section. So close i can taste it though. I really appreciate the help. I'm starting to wrap my mind around this stuff now, but doing it in QT is still pretty new to me.

[RobbieP]

The doubles are all coming in as 0 or very close to 0.

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

The Sender isn't written in QT...it's just using regular socket sending.

Whaaattt!!?? Why did you not say this from the outset? You cannot/must not/should not attempt to use QDataStream for anything other than QDataStream<->QDataStream exchange. It puts in its own meta-data.

Back to the drawing board. Remove QDataStream. Do whatever coding you need for whatever the other end sends in whatever format it uses....

If you insist on using QDataStream, just using raw byte stuff, I don't much see the point but it's up to you. Certainly then deserialize >> inPacket.Size; is pretty dodgy.

[RobbieP]

Sorry I thought I mentioned it...I looked back up and it seems I just said "sent in UDP" I meant to say it's using a custom udp sender and the packet is just sent as (char*)packet without going through a serializer. So...a little background information. The program I'm replacing with a QT app was multicast...it was destroying our network, but I don't have the source code for it. Only the source code for the sender. I've modified the sender (which is a plugin for the X-Plane flight simulator, so I can't use QT) to only do unicast. And I've rewritten the receiver in QT. So to be honest...I've never deserialized without qt. Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

Can I just read the data in directly to a struct and assume it would work if the byteorder is the same on both ends?

Nope, because of possible packing/data alignment issues....

At the end of the day you must know, somehow, how the data was sent/formatted. You will also need to know if it sends things like ints, and what endianness it does that in. As in: without that you can't read the data back sensibly.

I don't know whether others would say you can still use QDataStream::readRawData() safely, or if you did whether you are then getting much benefit out of QDataStream anyway.

If it were me I would just write code to read chunks of bytes as per whatever the sender's format is. Then I know where I am for sure. Qt/C++/C have functions to transform between endian integers without having to use QDataStream. Suggest do not read into a struct directly, read into each member separately as per what you were doing. Then alignment at receiving size won't be an issue. But without looking at sending code's side, can't be sure whether that might have padding/alignment. If we assume sender itself did not e.g. send any structs as a whole you may be good.

Off you go, and do lots of testing on sender's packets!

I've never deserialized without qt

OK, if you really want to try sticking with it. Just tell your receiver QDataStream to be little-endian. See whether that sorts the current issue to your satisfaction. If so, try lots of others. If they all work, you can stick with QDataStream I guess....

[RobbieP]

@JonB I'm getting the correct names and the size...the doubles are off though...does the second readRawData start where the previous readRawData left off? calling readRawData back to back like that doesn't reset the position does it?

[RobbieP]

@JonB I ask because I'm curious if it's possible the second readRawData isn't tryign to convert the 200 64 character names into doubles.

[JonB]

@RobbieP
No, your readRawData()s run sequentially, carrying on from where they got to.

Possibilities:

• Sender does have some pad between end of Name and start of Value.
• Sender saves doubles as something other than write(&double, sizeof(double)).
• Sender is different architecture which represents doubles differently. Though I kind of doubt this, I believe there is some standard for how doubles are represented which is cross-architecture.
• QDataStream encodes double with some extra information. Though I doubt this.

the doubles are off though

How "off" is "off"? Slightly wrong, or way out there?

Suggestions:

• You really should put those checks in on return result of QDataStream::readRawData().
• Can you get the sender to send just with inPacket.Size = 1 while you get it sorted out?
• Try it with plain read()s and no QDataStream. Though if that does not work you're a bit stuck.
• Do you have the source code of the sender's write()ing code to examine?

[RobbieP]

@JonB

/* This method writes data to the UDP socket and returns the number of
* bytes written or -1 for an error.  Pass in the IP and port youare sending to. */

int PCSBSocketUDP::WriteData(
				void *				inBuf,
				int				inBufLength,
				unsigned int		inDstIP,
				unsigned short		inDstPort)
{
	sockaddr_in destAddress;
	int tolen = sizeof(destAddress);
	destAddress.sin_family = AF_INET;
	destAddress.sin_port = htons(inDstPort);
	destAddress.sin_addr.s_addr = htonl(inDstIP);
	int result = sendto(mWinSocket,
						 (char*)inBuf,
						 inBufLength,
						 0,
						 (sockaddr*)&destAddress,
						 tolen);

	if (result == -1)
	{
		return -1;
	}
	else
	{
		return result;
	}
}

and I'm sending the struct from the top of my first post with that using:

m_SendSocket->WriteData((char *)&m_Packet, sizeof (fennecPacketStruct), ntohl(inet_addr(m_ipaddress.c_str())), 46456);

the sender is running on this same linux machine, but it's a plugin to a flight simulator. I'll see what the return value is for readRawData next.

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

m_SendSocket->WriteData((char *)&m_Packet, sizeof (fennecPacketStruct), ntohl(inet_addr(m_ipaddress.c_str())), 46456);

Since this sends a fennecPacketStruct in one chunk of bytes, why in the world are you trying to deserialize field-by-field? Assuming same machine architecture and same compiler and same definition of fennecPacketStruct with same constants for sizes, you want to just do the opposite read, like read(&inPacket, sizeof(fennecPacketStruct)).

Meanwhile I am "unconvinced" your sender size really has an identical declaration of struct fennecPacketStruct as you showed? Or does it? That uses a fixed size SP_PACKET_SIZE constant. But earlier your code read inPacket.Size from the data stream, and used that to affect how much it read where. Something is fishy...

[RobbieP]

That's what I was originally thinking a long time ago, but I can't seem to do it...it gives me this error:

/home/robbie/Documents/Programming/Fennec/mainwindow.cpp:36: error: no matching function for call to ‘QUdpSocket::read(MainWindow::fennecPacketStruct*, long unsigned int)’
../Fennec/mainwindow.cpp: In member function ‘void MainWindow::readPendingDatagrams()’:
../Fennec/mainwindow.cpp:36:24: error: no matching function for call to ‘QUdpSocket::read(MainWindow::fennecPacketStruct*, long unsigned int)’
36 | udpSocket->read(&inPacket, sizeof(fennecPacketStruct));
| ~^~~~~~~~~~~~~~~~~~~~~~~~~

[JonB]

@RobbieP
Of course that won't match. The writer code does not use fennecPacketStruct in WriteData() either. For qint64 QIODevice::read(char *data, qint64 maxSize) it expects a char *, so cast it....

[RobbieP]

@JonB the writer is using fennecPacketStruct though isn't it? m_Packet is a fennecPacketStruct, and the writer is sending sizeof(fennecPacketStruct).

Do I cast it in the read block using another reinterpret_cast? I'm not sure how that would work, because even casting it doesn't seem to like it. Also I need to use ReadDataGram, not Read because I need the host and port that it's coming from.

[RobbieP]

@JonB got it!

the correct cast was (char *)&inPacket

All my values are now correct!

I'll do some sanity checking and add some value logs as well to make sure this is goign to stay consistent in the lab. Thanks much for your incredible patience and help.

[JonB]

@RobbieP said in Still having trouble deserializing binary data from another UDP source (non-qt):

the correct cast was (char *)&inPacket

Yeah. Which should have gone through as reinterpret_cast<char *>(&inPacket) ? Not that it's much different, but keeps the wolves at bay.

the writer is using fennecPacketStruct though isn't it?

But not in the definition of PCSBSocketUDP::WriteData(void *inBuf, ...), or sendTo(..., (char*)inBuf), which are the equivalent of your udpSocket->read(&inPacket, ...).