Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Way to store large JSON
Forum Updated to NodeBB v4.3 + New Features

Way to store large JSON

Scheduled Pinned Locked Moved Solved General and Desktop
15 Posts 5 Posters 2.9k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • mrjjM Offline
    mrjjM Offline
    mrjj
    Lifetime Qt Champion
    wrote on last edited by
    #2

    Hi
    That must be crazy big.
    How many elements and how many attributes of of the objects
    are we talking about ?

    1 Reply Last reply
    0
    • E enne9

      Hello,
      in my application, I store objects in a QJsonArray.
      Here comes the problem: both QJsonObject in terms of maximum size and QJsonArray in terms of maximum number of elements seem to be limited.
      How to overcome this problem? What are the possible solutions?
      Thanks

      JonBJ Offline
      JonBJ Offline
      JonB
      wrote on last edited by
      #3

      @enne9 said in Way to store large JSON:

      Here comes the problem: both QJsonObject in terms of maximum size and QJsonArray in terms of maximum number of elements seem to be limited.

      Where do you get any evidence this is the case?

      Unless you mean limited to, say, 1 billion array elements, which wouldn't surprise me.

      1 Reply Last reply
      0
      • E Offline
        E Offline
        enne9
        wrote on last edited by
        #4

        I read that a QJsonObject has a maximum size of 128MB, but this is not much of a limitation in my case.
        My problem is the limited dimension of QJsonArray: I need to add many elements and after a while, it returns "QJson: Document too large to store in data structure".
        Do you know what is the maximum size of a QJsonArray? I wasn't able to find it in the documentation.

        1 Reply Last reply
        0
        • Christian EhrlicherC Offline
          Christian EhrlicherC Offline
          Christian Ehrlicher
          Lifetime Qt Champion
          wrote on last edited by
          #5

          @enne9 said in Way to store large JSON:

          I wasn't able to find it in the documentation.

          You already know it:

          I read that a QJsonObject has a maximum size of 128MB,

          Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
          Visit the Qt Academy at https://academy.qt.io/catalog

          1 Reply Last reply
          0
          • mrjjM Offline
            mrjjM Offline
            mrjj
            Lifetime Qt Champion
            wrote on last edited by mrjj
            #6

            Hi
            It might be fixed in Qt6
            https://bugreports.qt.io/browse/QTBUG-47629

            In the meantime, i can recommend
            https://github.com/nlohmann/json
            Using it at work with huge files.
            Not sure what actual limits are but i think its mostly memory
            constrains so for 64 bit apps. its truly many.

            1 Reply Last reply
            3
            • E Offline
              E Offline
              enne9
              wrote on last edited by
              #7

              Instead, if I want to split the QJsonArray once I reach the maximum size, how can I get the size? In terms of bytes, since the maximum is 128MB.

              So that I can generate and send multiple QJsonArray.

              mrjjM 1 Reply Last reply
              0
              • E enne9

                Instead, if I want to split the QJsonArray once I reach the maximum size, how can I get the size? In terms of bytes, since the maximum is 128MB.

                So that I can generate and send multiple QJsonArray.

                mrjjM Offline
                mrjjM Offline
                mrjj
                Lifetime Qt Champion
                wrote on last edited by
                #8

                @enne9
                Hi
                If you read the link they talk about it.
                You can use something like

                QJsonArray arr;
                
                // (...)
                
                int previousSize = arr.Size();
                
                arr.insert(someValue); // May be rejected
                
                if (arr.size() == previousSize) {
                
                throw SizeBeyondLimitEx();
                
                }
                

                To know when its time to start a new array.

                1 Reply Last reply
                1
                • E Offline
                  E Offline
                  enne9
                  wrote on last edited by
                  #9

                  But size() returns the number of elements, if I insert a new one it would always be greater than the previous one. Am I missing something?

                  mrjjM 1 Reply Last reply
                  0
                  • E enne9

                    But size() returns the number of elements, if I insert a new one it would always be greater than the previous one. Am I missing something?

                    mrjjM Offline
                    mrjjM Offline
                    mrjj
                    Lifetime Qt Champion
                    wrote on last edited by
                    #10

                    @enne9
                    Well the logic here is
                    if the size is the old size after you tried to insert new value its full/failed and you must start new one.

                    1 Reply Last reply
                    0
                    • E Offline
                      E Offline
                      enne9
                      wrote on last edited by
                      #11

                      Thanks, you're right.
                      In this case, do you know if there is any way to avoid the annoying output QJson: Document too large to store in data structure that I think will pop up once trying to insert after the limit is reached?

                      mrjjM 1 Reply Last reply
                      0
                      • E enne9

                        Thanks, you're right.
                        In this case, do you know if there is any way to avoid the annoying output QJson: Document too large to store in data structure that I think will pop up once trying to insert after the limit is reached?

                        mrjjM Offline
                        mrjjM Offline
                        mrjj
                        Lifetime Qt Champion
                        wrote on last edited by
                        #12

                        @enne9
                        Hi
                        I dont think so. (besides install message handler and simply remove it)
                        Else you would have to sum up all data strings to get an estimate of when you have used 128MB.
                        But im not sure that so clear cut to keep track of.

                        1 Reply Last reply
                        0
                        • E Offline
                          E Offline
                          enne9
                          wrote on last edited by
                          #13

                          Instead, if I have a QJsonObject (already partially filled) and I want to put a QByteArray inside it?
                          I convert the QByteArray into a QString using QString(bytearray.toBase64());, but I'd like to know if there is a way to understand how to split the QByteArray into partitions to fit inside the QJsonObject that has already some content.

                          QJsonObject message;
                          message["header"] = QStringLiteral("test");
                          message["content"] = QString(bytearray.toBase64());
                          

                          The problem in the code above is that the QByteArray can be too big and I want to split it into multiple messages.

                          Do you have any smart ideas about how to handle that? I've checked the documentation but I didn't find any easy method to deal with that.

                          jsulmJ 1 Reply Last reply
                          0
                          • E enne9

                            Instead, if I have a QJsonObject (already partially filled) and I want to put a QByteArray inside it?
                            I convert the QByteArray into a QString using QString(bytearray.toBase64());, but I'd like to know if there is a way to understand how to split the QByteArray into partitions to fit inside the QJsonObject that has already some content.

                            QJsonObject message;
                            message["header"] = QStringLiteral("test");
                            message["content"] = QString(bytearray.toBase64());
                            

                            The problem in the code above is that the QByteArray can be too big and I want to split it into multiple messages.

                            Do you have any smart ideas about how to handle that? I've checked the documentation but I didn't find any easy method to deal with that.

                            jsulmJ Offline
                            jsulmJ Offline
                            jsulm
                            Lifetime Qt Champion
                            wrote on last edited by
                            #14

                            @enne9 Why do you want to squeeze huge binary data into JSON?! JSON is a text format.
                            Why not store binary data in files and put files names into JSON document?

                            https://forum.qt.io/topic/113070/qt-code-of-conduct

                            1 Reply Last reply
                            2
                            • mrjjM Offline
                              mrjjM Offline
                              mrjj
                              Lifetime Qt Champion
                              wrote on last edited by
                              #15

                              @enne9

                              • Do you have any smart ideas about how to handle that?

                              Well based on experience, i would really look over the existing code and see how much effort it would be to convert the code to use
                              https://github.com/nlohmann/json

                              Why you may ask ?
                              Well, when if you have to make hacks/workarounds to use a certain technology, at some point
                              the cost/pain of maintaining/expanding such code often outweighs the cost of writing it again using a technology that removes the need for workarounds.

                              nlohmann json you would just store what you need. Should handle it easily.

                              That lib is super flexible so might require almost no code changes.

                              // create an empty structure (null)
                              json j;
                              
                              // add a number that is stored as double (note the implicit conversion of j to an object)
                              j["pi"] = 3.141;
                              
                              // add a Boolean that is stored as bool
                              j["happy"] = true;
                              
                              // add a string that is stored as std::string
                              j["name"] = "Niels";
                              
                              // add another null object by passing nullptr
                              j["nothing"] = nullptr;
                              
                              // add an object inside the object
                              j["answer"]["everything"] = 42;
                              
                              // add an array that is stored as std::vector (using an initializer list)
                              j["list"] = { 1, 0, 2 };
                              
                              // add another object (using an initializer list of pairs)
                              j["object"] = { {"currency", "USD"}, {"value", 42.99} };
                              
                              // instead, you could also write (which looks very similar to the JSON above)
                              json j2 = {
                                {"pi", 3.141},
                                {"happy", true},
                                {"name", "Niels"},
                                {"nothing", nullptr},
                                {"answer", {
                                  {"everything", 42}
                                }},
                                {"list", {1, 0, 2}},
                                {"object", {
                                  {"currency", "USD"},
                                  {"value", 42.99}
                                }}
                              };
                              
                              1 Reply Last reply
                              0

                              • Login

                              • Login or register to search.
                              • First post
                                Last post
                              0
                              • Categories
                              • Recent
                              • Tags
                              • Popular
                              • Users
                              • Groups
                              • Search
                              • Get Qt Extensions
                              • Unsolved