What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?

Mike Fidel

Greetings!

I would like to know if there is a significant performance\size cost to serializing a QMap with QDataStream as opposed to converting the QMap into a QList or a regular QString (For example by converting the QMap into a comma separated values QString), and serializing them.

JonB

@Mike-Fidel
One comment about the "performance" (assuming you mean "speed", and even "space") side: do you mean your alternative is to convert the QMap "on-the-fly" just in order to serialize it, or do you mean you would maintain the corresponding list/string already? My thought is that converting it to whatever just to serialize it would itself likely cost you as much time/space as whatever you would save during the subsequent serialization.

Another separate point is whether you need/already have to have a QMap rather than, say, a QHash?

Mike Fidel

@JonB

Do you mean your alternative is to convert the QMap "on-the-fly" just in order to serialize it, or do you mean you would maintain the corresponding list/string already.

I mean both. What is the performance impact in converting the QMap on the the fly and then serializing the result as opposed to serializing the QMap as is. And Also I mean what is the cost of serializing the QMap as is Versus serializing an already converted QString.

In other words, is serialization of the QMap as is, is significantly more costly in terms of size and speed as opposed to converting the QMap to a string, and also as opposed to serializing a pre-converted QString.

Another separate point is whether you need/already have to have a QMap rather than, say, a QHash?

I don't have a QMap already, because I have not yet decided on what data stricture to use. I would assume that serialization of QHash is more costly than QMap.

JonB

@Mike-Fidel
I am not a Qt expert, unlike some.

My observation about "on-the-fly" was that, intuitively to me, I would be surprised if first converting to another data representation just in order to serialize would end up saving time or space --- I would have thought the opposite.

As for QHash vs QMap. I assume you must have a large number of elements to care about serialization performance. Have a read of https://woboq.com/blog/qmap_qhash_benchmark.html. It is not to do with serialization, but might well affect your usage. And also note the conclusion:

The typical rule is: Use QMap only if you need the items to be sorted or if you know that you always have a very small amount of items in your map.

kshegunov

@Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

I would like to know if there is a significant performance\size cost to serializing a QMap with QDataStream as opposed to converting the QMap into a QList or a regular QString

How would you convert a key-value container to a sequential one (i.e. QList)? And also QList is a bad choice if the data is larger than the size of void *.

What is the performance impact in converting the QMap on the the fly and then serializing the result as opposed to serializing the QMap as is.

The conversion cost for the key and value into the temporary + the serialization cost of the intermediate result. With great probability the conversion cost to an intermediate data structure will overgrow the serialization cost of the key-value pairs significantly for all possible cases.

And Also I mean what is the cost of serializing the QMap as is Versus serializing an already converted QString.

I don't see how serializing to text as intermediary lifts from from the weight at all. Moreover deserializing from the intermediate results will really sink your ship ...

In other words, is serialization of the QMap as is, is significantly more costly in terms of size and speed as opposed to converting the QMap to a string

Almost certainly not.

and also as opposed to serializing a pre-converted QString

If you need to construct the pre-converted string which I imagine you do, same answer as above.

I would assume that serialization of QHash is more costly than QMap.

Why would you assume that?

Mike Fidel

@kshegunov said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

How would you convert a key-value container to a sequential one (i.e. QList)?

For example I could put them in a string ''{Key: Value}'' if the QMap is <QString,QString>.

I would assume that serialization of QHash is more costly than QMap.

Why would you assume that?

Because QHash needs to also store the 'hash' value for each key.

kshegunov

@Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

For example I could put them in a string ''{Key: Value}'' if the QMap is <QString,QString>.

Which is already serializing the data to a text stream. You just would do it two times for no obviously identifiable reason.

Because QHash needs to also store the 'hash' value for each key.

No it doesn't. The hash table stores the key (to handle hash collisions) and the value in each bucket. There's no reason to store the hash, as it's an implementation detail that is not required to be persistent. If you access by key, the key is hashed on the fly to find the bucket. If you use iterators, then the hash is not at all needed.

Mike Fidel

@kshegunov

Thanks! This is very informative.

If you need to construct the pre-converted string which I imagine you do, same answer as above.

What if I don't need to pre-convert the string. If I already have the string for 'free'...

Let's say I have a QMap and a QString with the same amount of data. What would be the penalty for serializing QMap vs QString?

I guess what I'm asking is whether or not serializing QMap (Or QHash) incur significant overhead when compared to serializing the same amount of data in more basic types like QString.

kshegunov

@Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

Let's say I have a QMap and a QString with the same amount of data. What would be the penalty for serializing QMap vs QString?

This is really hard to answer, I'd run a series of tests if I really cared about it.

I guess what I'm asking is whether or not serializing QMap (Or QHash) incur significant overhead when compared to serializing the same amount of data in more basic types like QString.

As the containers are templates it will vary between the used key and value types, but if we assume they are QStrings, I would put an educated guess and say, no not significant.

JonB

@Mike-Fidel
One thing: do you have, say, tens of thousands of keys in your QMap?

Mike Fidel

@JonB No... No more than about10 key value pairs.

kshegunov

Then what's the relevance of how efficient serialization is ...?

JonB

@kshegunov Hence my question :)

JKSH

@Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

No more than about10 key value pairs.

With such a small amount of data, the "performance, space and overhead" really doesn't matter (unless you're serializing and deserializing hundreds of thousands of times a second...?)

Choose the option that gives you the most readable code.

Related reading: https://stackify.com/premature-optimization-evil/

I would like to know if there is a significant performance\size cost to...

The best way to find out is to benchmark them.

JonB

@JKSH said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

With such a small amount of data, the "performance, space and overhead" really doesn't matter (unless you're serializing and deserializing hundreds of thousands of times a second...?)

And if you were serializing the same QMap(s) hundreds of thousands times per second, you should rather look a caching the result(s) of the serialization/deserialization than at the serialization code itself, of course!

Mike Fidel

Thank you everyone for your answers!