Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?
Forum Updated to NodeBB v4.3 + New Features

What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?

Scheduled Pinned Locked Moved Solved General and Desktop
16 Posts 4 Posters 3.0k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M Offline
    M Offline
    Mike Fidel
    wrote on last edited by
    #1

    Greetings!

    I would like to know if there is a significant performance\size cost to serializing a QMap with QDataStream as opposed to converting the QMap into a QList or a regular QString (For example by converting the QMap into a comma separated values QString), and serializing them.

    JonBJ 1 Reply Last reply
    0
    • M Mike Fidel

      Greetings!

      I would like to know if there is a significant performance\size cost to serializing a QMap with QDataStream as opposed to converting the QMap into a QList or a regular QString (For example by converting the QMap into a comma separated values QString), and serializing them.

      JonBJ Offline
      JonBJ Offline
      JonB
      wrote on last edited by JonB
      #2

      @Mike-Fidel
      One comment about the "performance" (assuming you mean "speed", and even "space") side: do you mean your alternative is to convert the QMap "on-the-fly" just in order to serialize it, or do you mean you would maintain the corresponding list/string already? My thought is that converting it to whatever just to serialize it would itself likely cost you as much time/space as whatever you would save during the subsequent serialization.

      Another separate point is whether you need/already have to have a QMap rather than, say, a QHash?

      M 1 Reply Last reply
      1
      • JonBJ JonB

        @Mike-Fidel
        One comment about the "performance" (assuming you mean "speed", and even "space") side: do you mean your alternative is to convert the QMap "on-the-fly" just in order to serialize it, or do you mean you would maintain the corresponding list/string already? My thought is that converting it to whatever just to serialize it would itself likely cost you as much time/space as whatever you would save during the subsequent serialization.

        Another separate point is whether you need/already have to have a QMap rather than, say, a QHash?

        M Offline
        M Offline
        Mike Fidel
        wrote on last edited by
        #3

        @JonB

        Do you mean your alternative is to convert the QMap "on-the-fly" just in order to serialize it, or do you mean you would maintain the corresponding list/string already.

        I mean both. What is the performance impact in converting the QMap on the the fly and then serializing the result as opposed to serializing the QMap as is. And Also I mean what is the cost of serializing the QMap as is Versus serializing an already converted QString.

        In other words, is serialization of the QMap as is, is significantly more costly in terms of size and speed as opposed to converting the QMap to a string, and also as opposed to serializing a pre-converted QString.

        Another separate point is whether you need/already have to have a QMap rather than, say, a QHash?

        I don't have a QMap already, because I have not yet decided on what data stricture to use. I would assume that serialization of QHash is more costly than QMap.

        JonBJ kshegunovK 2 Replies Last reply
        0
        • M Mike Fidel

          @JonB

          Do you mean your alternative is to convert the QMap "on-the-fly" just in order to serialize it, or do you mean you would maintain the corresponding list/string already.

          I mean both. What is the performance impact in converting the QMap on the the fly and then serializing the result as opposed to serializing the QMap as is. And Also I mean what is the cost of serializing the QMap as is Versus serializing an already converted QString.

          In other words, is serialization of the QMap as is, is significantly more costly in terms of size and speed as opposed to converting the QMap to a string, and also as opposed to serializing a pre-converted QString.

          Another separate point is whether you need/already have to have a QMap rather than, say, a QHash?

          I don't have a QMap already, because I have not yet decided on what data stricture to use. I would assume that serialization of QHash is more costly than QMap.

          JonBJ Offline
          JonBJ Offline
          JonB
          wrote on last edited by
          #4

          @Mike-Fidel
          I am not a Qt expert, unlike some.

          My observation about "on-the-fly" was that, intuitively to me, I would be surprised if first converting to another data representation just in order to serialize would end up saving time or space --- I would have thought the opposite.

          As for QHash vs QMap. I assume you must have a large number of elements to care about serialization performance. Have a read of https://woboq.com/blog/qmap_qhash_benchmark.html. It is not to do with serialization, but might well affect your usage. And also note the conclusion:

          The typical rule is: Use QMap only if you need the items to be sorted or if you know that you always have a very small amount of items in your map.

          1 Reply Last reply
          2
          • M Mike Fidel

            @JonB

            Do you mean your alternative is to convert the QMap "on-the-fly" just in order to serialize it, or do you mean you would maintain the corresponding list/string already.

            I mean both. What is the performance impact in converting the QMap on the the fly and then serializing the result as opposed to serializing the QMap as is. And Also I mean what is the cost of serializing the QMap as is Versus serializing an already converted QString.

            In other words, is serialization of the QMap as is, is significantly more costly in terms of size and speed as opposed to converting the QMap to a string, and also as opposed to serializing a pre-converted QString.

            Another separate point is whether you need/already have to have a QMap rather than, say, a QHash?

            I don't have a QMap already, because I have not yet decided on what data stricture to use. I would assume that serialization of QHash is more costly than QMap.

            kshegunovK Offline
            kshegunovK Offline
            kshegunov
            Moderators
            wrote on last edited by kshegunov
            #5

            @Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

            I would like to know if there is a significant performance\size cost to serializing a QMap with QDataStream as opposed to converting the QMap into a QList or a regular QString

            How would you convert a key-value container to a sequential one (i.e. QList)? And also QList is a bad choice if the data is larger than the size of void *.

            What is the performance impact in converting the QMap on the the fly and then serializing the result as opposed to serializing the QMap as is.

            The conversion cost for the key and value into the temporary + the serialization cost of the intermediate result. With great probability the conversion cost to an intermediate data structure will overgrow the serialization cost of the key-value pairs significantly for all possible cases.

            And Also I mean what is the cost of serializing the QMap as is Versus serializing an already converted QString.

            I don't see how serializing to text as intermediary lifts from from the weight at all. Moreover deserializing from the intermediate results will really sink your ship ...

            In other words, is serialization of the QMap as is, is significantly more costly in terms of size and speed as opposed to converting the QMap to a string

            Almost certainly not.

            and also as opposed to serializing a pre-converted QString

            If you need to construct the pre-converted string which I imagine you do, same answer as above.

            I would assume that serialization of QHash is more costly than QMap.

            Why would you assume that?

            Read and abide by the Qt Code of Conduct

            M 1 Reply Last reply
            3
            • kshegunovK kshegunov

              @Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

              I would like to know if there is a significant performance\size cost to serializing a QMap with QDataStream as opposed to converting the QMap into a QList or a regular QString

              How would you convert a key-value container to a sequential one (i.e. QList)? And also QList is a bad choice if the data is larger than the size of void *.

              What is the performance impact in converting the QMap on the the fly and then serializing the result as opposed to serializing the QMap as is.

              The conversion cost for the key and value into the temporary + the serialization cost of the intermediate result. With great probability the conversion cost to an intermediate data structure will overgrow the serialization cost of the key-value pairs significantly for all possible cases.

              And Also I mean what is the cost of serializing the QMap as is Versus serializing an already converted QString.

              I don't see how serializing to text as intermediary lifts from from the weight at all. Moreover deserializing from the intermediate results will really sink your ship ...

              In other words, is serialization of the QMap as is, is significantly more costly in terms of size and speed as opposed to converting the QMap to a string

              Almost certainly not.

              and also as opposed to serializing a pre-converted QString

              If you need to construct the pre-converted string which I imagine you do, same answer as above.

              I would assume that serialization of QHash is more costly than QMap.

              Why would you assume that?

              M Offline
              M Offline
              Mike Fidel
              wrote on last edited by
              #6

              @kshegunov said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

              How would you convert a key-value container to a sequential one (i.e. QList)?

              For example I could put them in a string ''{Key: Value}'' if the QMap is <QString,QString>.

              I would assume that serialization of QHash is more costly than QMap.

              Why would you assume that?

              Because QHash needs to also store the 'hash' value for each key.

              kshegunovK 1 Reply Last reply
              0
              • M Mike Fidel

                @kshegunov said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

                How would you convert a key-value container to a sequential one (i.e. QList)?

                For example I could put them in a string ''{Key: Value}'' if the QMap is <QString,QString>.

                I would assume that serialization of QHash is more costly than QMap.

                Why would you assume that?

                Because QHash needs to also store the 'hash' value for each key.

                kshegunovK Offline
                kshegunovK Offline
                kshegunov
                Moderators
                wrote on last edited by
                #7

                @Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

                For example I could put them in a string ''{Key: Value}'' if the QMap is <QString,QString>.

                Which is already serializing the data to a text stream. You just would do it two times for no obviously identifiable reason.

                Because QHash needs to also store the 'hash' value for each key.

                No it doesn't. The hash table stores the key (to handle hash collisions) and the value in each bucket. There's no reason to store the hash, as it's an implementation detail that is not required to be persistent. If you access by key, the key is hashed on the fly to find the bucket. If you use iterators, then the hash is not at all needed.

                Read and abide by the Qt Code of Conduct

                M 1 Reply Last reply
                3
                • kshegunovK kshegunov

                  @Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

                  For example I could put them in a string ''{Key: Value}'' if the QMap is <QString,QString>.

                  Which is already serializing the data to a text stream. You just would do it two times for no obviously identifiable reason.

                  Because QHash needs to also store the 'hash' value for each key.

                  No it doesn't. The hash table stores the key (to handle hash collisions) and the value in each bucket. There's no reason to store the hash, as it's an implementation detail that is not required to be persistent. If you access by key, the key is hashed on the fly to find the bucket. If you use iterators, then the hash is not at all needed.

                  M Offline
                  M Offline
                  Mike Fidel
                  wrote on last edited by
                  #8

                  @kshegunov

                  Thanks! This is very informative.

                  If you need to construct the pre-converted string which I imagine you do, same answer as above.

                  What if I don't need to pre-convert the string. If I already have the string for 'free'...

                  Let's say I have a QMap and a QString with the same amount of data. What would be the penalty for serializing QMap vs QString?

                  I guess what I'm asking is whether or not serializing QMap (Or QHash) incur significant overhead when compared to serializing the same amount of data in more basic types like QString.

                  kshegunovK JonBJ 2 Replies Last reply
                  0
                  • M Mike Fidel

                    @kshegunov

                    Thanks! This is very informative.

                    If you need to construct the pre-converted string which I imagine you do, same answer as above.

                    What if I don't need to pre-convert the string. If I already have the string for 'free'...

                    Let's say I have a QMap and a QString with the same amount of data. What would be the penalty for serializing QMap vs QString?

                    I guess what I'm asking is whether or not serializing QMap (Or QHash) incur significant overhead when compared to serializing the same amount of data in more basic types like QString.

                    kshegunovK Offline
                    kshegunovK Offline
                    kshegunov
                    Moderators
                    wrote on last edited by kshegunov
                    #9

                    @Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

                    Let's say I have a QMap and a QString with the same amount of data. What would be the penalty for serializing QMap vs QString?

                    This is really hard to answer, I'd run a series of tests if I really cared about it.

                    I guess what I'm asking is whether or not serializing QMap (Or QHash) incur significant overhead when compared to serializing the same amount of data in more basic types like QString.

                    As the containers are templates it will vary between the used key and value types, but if we assume they are QStrings, I would put an educated guess and say, no not significant.

                    Read and abide by the Qt Code of Conduct

                    1 Reply Last reply
                    1
                    • M Mike Fidel

                      @kshegunov

                      Thanks! This is very informative.

                      If you need to construct the pre-converted string which I imagine you do, same answer as above.

                      What if I don't need to pre-convert the string. If I already have the string for 'free'...

                      Let's say I have a QMap and a QString with the same amount of data. What would be the penalty for serializing QMap vs QString?

                      I guess what I'm asking is whether or not serializing QMap (Or QHash) incur significant overhead when compared to serializing the same amount of data in more basic types like QString.

                      JonBJ Offline
                      JonBJ Offline
                      JonB
                      wrote on last edited by
                      #10

                      @Mike-Fidel
                      One thing: do you have, say, tens of thousands of keys in your QMap?

                      M 1 Reply Last reply
                      0
                      • JonBJ JonB

                        @Mike-Fidel
                        One thing: do you have, say, tens of thousands of keys in your QMap?

                        M Offline
                        M Offline
                        Mike Fidel
                        wrote on last edited by Mike Fidel
                        #11

                        @JonB No... No more than about10 key value pairs.

                        kshegunovK JKSHJ 2 Replies Last reply
                        0
                        • M Mike Fidel

                          @JonB No... No more than about10 key value pairs.

                          kshegunovK Offline
                          kshegunovK Offline
                          kshegunov
                          Moderators
                          wrote on last edited by
                          #12

                          Then what's the relevance of how efficient serialization is ...?

                          Read and abide by the Qt Code of Conduct

                          JonBJ 1 Reply Last reply
                          3
                          • kshegunovK kshegunov

                            Then what's the relevance of how efficient serialization is ...?

                            JonBJ Offline
                            JonBJ Offline
                            JonB
                            wrote on last edited by
                            #13

                            @kshegunov Hence my question :)

                            1 Reply Last reply
                            1
                            • M Mike Fidel

                              @JonB No... No more than about10 key value pairs.

                              JKSHJ Offline
                              JKSHJ Offline
                              JKSH
                              Moderators
                              wrote on last edited by JKSH
                              #14

                              @Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

                              No more than about10 key value pairs.

                              With such a small amount of data, the "performance, space and overhead" really doesn't matter (unless you're serializing and deserializing hundreds of thousands of times a second...?)

                              Choose the option that gives you the most readable code.

                              Related reading: https://stackify.com/premature-optimization-evil/

                              I would like to know if there is a significant performance\size cost to...

                              The best way to find out is to benchmark them.

                              Qt Doc Search for browsers: forum.qt.io/topic/35616/web-browser-extension-for-improved-doc-searches

                              JonBJ 1 Reply Last reply
                              0
                              • JKSHJ JKSH

                                @Mike-Fidel said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

                                No more than about10 key value pairs.

                                With such a small amount of data, the "performance, space and overhead" really doesn't matter (unless you're serializing and deserializing hundreds of thousands of times a second...?)

                                Choose the option that gives you the most readable code.

                                Related reading: https://stackify.com/premature-optimization-evil/

                                I would like to know if there is a significant performance\size cost to...

                                The best way to find out is to benchmark them.

                                JonBJ Offline
                                JonBJ Offline
                                JonB
                                wrote on last edited by
                                #15

                                @JKSH said in What is the serialization cost of QMap VS QList or QString in terms of performance, space and overhead?:

                                With such a small amount of data, the "performance, space and overhead" really doesn't matter (unless you're serializing and deserializing hundreds of thousands of times a second...?)

                                And if you were serializing the same QMap(s) hundreds of thousands times per second, you should rather look a caching the result(s) of the serialization/deserialization than at the serialization code itself, of course!

                                1 Reply Last reply
                                0
                                • M Offline
                                  M Offline
                                  Mike Fidel
                                  wrote on last edited by
                                  #16

                                  Thank you everyone for your answers!

                                  1 Reply Last reply
                                  0

                                  • Login

                                  • Login or register to search.
                                  • First post
                                    Last post
                                  0
                                  • Categories
                                  • Recent
                                  • Tags
                                  • Popular
                                  • Users
                                  • Groups
                                  • Search
                                  • Get Qt Extensions
                                  • Unsolved