Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

Way to store large JSON



  • Hello,
    in my application, I store objects in a QJsonArray.
    Here comes the problem: both QJsonObject in terms of maximum size and QJsonArray in terms of maximum number of elements seem to be limited.
    How to overcome this problem? What are the possible solutions?
    Thanks


  • Lifetime Qt Champion

    Hi
    That must be crazy big.
    How many elements and how many attributes of of the objects
    are we talking about ?



  • @enne9 said in Way to store large JSON:

    Here comes the problem: both QJsonObject in terms of maximum size and QJsonArray in terms of maximum number of elements seem to be limited.

    Where do you get any evidence this is the case?

    Unless you mean limited to, say, 1 billion array elements, which wouldn't surprise me.



  • I read that a QJsonObject has a maximum size of 128MB, but this is not much of a limitation in my case.
    My problem is the limited dimension of QJsonArray: I need to add many elements and after a while, it returns "QJson: Document too large to store in data structure".
    Do you know what is the maximum size of a QJsonArray? I wasn't able to find it in the documentation.


  • Qt Champions 2019

    @enne9 said in Way to store large JSON:

    I wasn't able to find it in the documentation.

    You already know it:

    I read that a QJsonObject has a maximum size of 128MB,


  • Lifetime Qt Champion

    Hi
    It might be fixed in Qt6
    https://bugreports.qt.io/browse/QTBUG-47629

    In the meantime, i can recommend
    https://github.com/nlohmann/json
    Using it at work with huge files.
    Not sure what actual limits are but i think its mostly memory
    constrains so for 64 bit apps. its truly many.



  • Instead, if I want to split the QJsonArray once I reach the maximum size, how can I get the size? In terms of bytes, since the maximum is 128MB.

    So that I can generate and send multiple QJsonArray.


  • Lifetime Qt Champion

    @enne9
    Hi
    If you read the link they talk about it.
    You can use something like

    QJsonArray arr;
    
    // (...)
    
    int previousSize = arr.Size();
    
    arr.insert(someValue); // May be rejected
    
    if (arr.size() == previousSize) {
    
    throw SizeBeyondLimitEx();
    
    }
    

    To know when its time to start a new array.



  • But size() returns the number of elements, if I insert a new one it would always be greater than the previous one. Am I missing something?


  • Lifetime Qt Champion

    @enne9
    Well the logic here is
    if the size is the old size after you tried to insert new value its full/failed and you must start new one.



  • Thanks, you're right.
    In this case, do you know if there is any way to avoid the annoying output QJson: Document too large to store in data structure that I think will pop up once trying to insert after the limit is reached?


  • Lifetime Qt Champion

    @enne9
    Hi
    I dont think so. (besides install message handler and simply remove it)
    Else you would have to sum up all data strings to get an estimate of when you have used 128MB.
    But im not sure that so clear cut to keep track of.



  • Instead, if I have a QJsonObject (already partially filled) and I want to put a QByteArray inside it?
    I convert the QByteArray into a QString using QString(bytearray.toBase64());, but I'd like to know if there is a way to understand how to split the QByteArray into partitions to fit inside the QJsonObject that has already some content.

    QJsonObject message;
    message["header"] = QStringLiteral("test");
    message["content"] = QString(bytearray.toBase64());
    

    The problem in the code above is that the QByteArray can be too big and I want to split it into multiple messages.

    Do you have any smart ideas about how to handle that? I've checked the documentation but I didn't find any easy method to deal with that.


  • Qt Champions 2019

    @enne9 Why do you want to squeeze huge binary data into JSON?! JSON is a text format.
    Why not store binary data in files and put files names into JSON document?


  • Lifetime Qt Champion

    @enne9

    • Do you have any smart ideas about how to handle that?

    Well based on experience, i would really look over the existing code and see how much effort it would be to convert the code to use
    https://github.com/nlohmann/json

    Why you may ask ?
    Well, when if you have to make hacks/workarounds to use a certain technology, at some point
    the cost/pain of maintaining/expanding such code often outweighs the cost of writing it again using a technology that removes the need for workarounds.

    nlohmann json you would just store what you need. Should handle it easily.

    That lib is super flexible so might require almost no code changes.

    // create an empty structure (null)
    json j;
    
    // add a number that is stored as double (note the implicit conversion of j to an object)
    j["pi"] = 3.141;
    
    // add a Boolean that is stored as bool
    j["happy"] = true;
    
    // add a string that is stored as std::string
    j["name"] = "Niels";
    
    // add another null object by passing nullptr
    j["nothing"] = nullptr;
    
    // add an object inside the object
    j["answer"]["everything"] = 42;
    
    // add an array that is stored as std::vector (using an initializer list)
    j["list"] = { 1, 0, 2 };
    
    // add another object (using an initializer list of pairs)
    j["object"] = { {"currency", "USD"}, {"value", 42.99} };
    
    // instead, you could also write (which looks very similar to the JSON above)
    json j2 = {
      {"pi", 3.141},
      {"happy", true},
      {"name", "Niels"},
      {"nothing", nullptr},
      {"answer", {
        {"everything", 42}
      }},
      {"list", {1, 0, 2}},
      {"object", {
        {"currency", "USD"},
        {"value", 42.99}
      }}
    };
    

Log in to reply