QDataStream serialization of objects over QTcpSocket
-
I am exploring using QDataStream serialization of objects over a TCP socket. I see many examples which use files, but it is unclear to me how this extends to socket transfer. In the examples I see, the authors typically do not send size information over the stream. This works fine for files, but with sockets, I can't guarantee all of the data needed to deserialize an object has been transmitted. How then do I ensure that I have complete data prior to attempting deserialization? Do I have to figure out the serialized sizes of all my data objects and encode it on the stream? Are there any examples of serializing user-defined objects over TCP socket using QDataStream?
-
@DRoscoe said:
QDataStream
When streaming something the actual format is always open.
One options is to use a fixed size header so that you will know when u have
received the expected amount. this header could then define a size for
the rest of the data or define a end of transmission byte.
Other options include serialize to json and send that over the wire.
If u stream "raw" object members then extra care for strings or composite types
must be handled in both ends.
Pointers also pose a case. -
@DRoscoe said:
QDataStream
When streaming something the actual format is always open.
One options is to use a fixed size header so that you will know when u have
received the expected amount. this header could then define a size for
the rest of the data or define a end of transmission byte.
Other options include serialize to json and send that over the wire.
If u stream "raw" object members then extra care for strings or composite types
must be handled in both ends.
Pointers also pose a case.@mrjj I was afraid of that. This is for a messaging system where we have our main application sending messages to a client. Currently, we have a home-grown XML-based serialization/deserialization approach which flattens out our data structures to text for transmission and reconstitution on the other side. This makes it easy to determine total xfer size, but its also inefficient. If I were to use JSON, then I'm recreating our current approach with one that is very similar. I am hoping to eventually supplant our home-grown XML-based serialization/deserialization approach with a Qt solution.
In any case, you've answered my question. Since we are using a fixed-size header which includes the total bytes expected, I could use the QDataStream to tell me the bytes available after serialization and backfill that information in the header. I just wasn't sure there wasn't a different approach since I could not find anything specific to serialization over a socket.
-
well if u already use XML then json would/might only decrease the traffic some % and maybe be slightly lighter depending on the xml lib used.
Is the current XML implementation too slow ?
-
well if u already use XML then json would/might only decrease the traffic some % and maybe be slightly lighter depending on the xml lib used.
Is the current XML implementation too slow ?
@mrjj I have a few reasons for looking into this. The libraries we use for messaging bring in a lot of dependencies that prevent it from being convenient for other consumers (apps) to use. If I were to switch to a Qt-centric approach using QDataStream, I could reduce this dependency to a small library defining the common elements for messaging, and then allow anyone who wants to create/send/receive messages to define their own, without incurring all of the current dependency overhead. From what I am reading, the QDataStream encodes most data types as binary, which would be more efficient with respect to traffic, and finally, our XML-based approach is very inefficient in its own right and somewhat clunky to use for anything but the simplest of class objects. I would prefer replacing it with a backbone that is community updated, maintained, optimized and debugged than keeping our home-grown approach
-
Hi,
Depending on your needs QtRemoteObjects might be of interest.
What about solutions like protobuf ?
-
@mrjj I have a few reasons for looking into this. The libraries we use for messaging bring in a lot of dependencies that prevent it from being convenient for other consumers (apps) to use. If I were to switch to a Qt-centric approach using QDataStream, I could reduce this dependency to a small library defining the common elements for messaging, and then allow anyone who wants to create/send/receive messages to define their own, without incurring all of the current dependency overhead. From what I am reading, the QDataStream encodes most data types as binary, which would be more efficient with respect to traffic, and finally, our XML-based approach is very inefficient in its own right and somewhat clunky to use for anything but the simplest of class objects. I would prefer replacing it with a backbone that is community updated, maintained, optimized and debugged than keeping our home-grown approach
@DRoscoe
There are two approaches to this:- Develop a protocol (this entails having special sequences to mark begin/end of message) and parsing them on reception. Usually it's a cumbersome approach to do if you're only interested to send a few objects. Which leads to:
- Send size information before the message so you know how much data you expect at the other end.
As for the serialization/deserialization you can look up here and here. Also note that sending binary data over network when running on different platforms implies care for endianness and for the version of QDataStream.
If you're interested in implementing it yourself, you could also peak at a project of mine (hadn't had the time to finish it properly, but it's a start) which deals with exactly such a problem, only the medium is a bit different but this is of no consequence here.
Of course what @SGaist suggested is also a possibility if you are willing to allow for the additional dependency (which I assume you are, since you say currently you have more than one).
Kind regards.
-
@DRoscoe
There are two approaches to this:- Develop a protocol (this entails having special sequences to mark begin/end of message) and parsing them on reception. Usually it's a cumbersome approach to do if you're only interested to send a few objects. Which leads to:
- Send size information before the message so you know how much data you expect at the other end.
As for the serialization/deserialization you can look up here and here. Also note that sending binary data over network when running on different platforms implies care for endianness and for the version of QDataStream.
If you're interested in implementing it yourself, you could also peak at a project of mine (hadn't had the time to finish it properly, but it's a start) which deals with exactly such a problem, only the medium is a bit different but this is of no consequence here.
Of course what @SGaist suggested is also a possibility if you are willing to allow for the additional dependency (which I assume you are, since you say currently you have more than one).
Kind regards.
@kshegunov said:
Also note that sending binary data over network when running on different platforms implies care for endianness and for the version of QDataStream.
I thought QDataStream took care of normalizing endianness. If I use the default value for the sender and all clients, I should be fine regardless of platform, correct? I understand about the versioning of the QDataStream.
Thanks for the link to your project. I will definitely be checking it out.
I think sending size information as part of the message header is the way to go. It's nearly impossible to use a single-character EOT for binary data, and using multi-character EOT sequence seems like overkill when a header can suffice
-
Hi,
Depending on your needs QtRemoteObjects might be of interest.
What about solutions like protobuf ?
-
@kshegunov said:
Also note that sending binary data over network when running on different platforms implies care for endianness and for the version of QDataStream.
I thought QDataStream took care of normalizing endianness. If I use the default value for the sender and all clients, I should be fine regardless of platform, correct? I understand about the versioning of the QDataStream.
Thanks for the link to your project. I will definitely be checking it out.
I think sending size information as part of the message header is the way to go. It's nearly impossible to use a single-character EOT for binary data, and using multi-character EOT sequence seems like overkill when a header can suffice
I thought QDataStream took care of normalizing endianness. If I use the default value for the sender and all clients, I should be fine regardless of platform, correct? I understand about the versioning of the QDataStream.
Hm, yes, I must have remembered wrong about the default setting. I thought it's platform dependent, but the docs clearly state it isn't. So you're absolutely fine not worrying about it.
I think sending size information as part of the message header is the way to go. It's nearly impossible to use a single-character EOT for binary data, and using multi-character EOT sequence seems like overkill when a header can suffice
I agree. Most of the time you don't need a full-fledged protocol and the hassle of implementing such thing for simple binary transfers is ... well, unwarranted. Plus it's not such a great improvement of using a protocol when you couple reception with parsing, and if you don't want to couple them, there's a whole new abstraction layer to be made ensuring the protocol is independent of the media ...
Kind regards.
