Way to store large JSON

enne9

Hello,
in my application, I store objects in a QJsonArray.
Here comes the problem: both QJsonObject in terms of maximum size and QJsonArray in terms of maximum number of elements seem to be limited.
How to overcome this problem? What are the possible solutions?
Thanks

mrjj

Hi
That must be crazy big.
How many elements and how many attributes of of the objects
are we talking about ?

JonB

@enne9 said in Way to store large JSON:

Here comes the problem: both QJsonObject in terms of maximum size and QJsonArray in terms of maximum number of elements seem to be limited.

Where do you get any evidence this is the case?

Unless you mean limited to, say, 1 billion array elements, which wouldn't surprise me.

enne9

I read that a QJsonObject has a maximum size of 128MB, but this is not much of a limitation in my case.
My problem is the limited dimension of QJsonArray: I need to add many elements and after a while, it returns "QJson: Document too large to store in data structure".
Do you know what is the maximum size of a QJsonArray? I wasn't able to find it in the documentation.

Christian Ehrlicher

@enne9 said in Way to store large JSON:

I wasn't able to find it in the documentation.

You already know it:

I read that a QJsonObject has a maximum size of 128MB,

mrjj

Hi
It might be fixed in Qt6
https://bugreports.qt.io/browse/QTBUG-47629

In the meantime, i can recommend
https://github.com/nlohmann/json
Using it at work with huge files.
Not sure what actual limits are but i think its mostly memory
constrains so for 64 bit apps. its truly many.

enne9

Instead, if I want to split the QJsonArray once I reach the maximum size, how can I get the size? In terms of bytes, since the maximum is 128MB.

So that I can generate and send multiple QJsonArray.

mrjj

@enne9
Hi
If you read the link they talk about it.
You can use something like

QJsonArray arr;

// (...)

int previousSize = arr.Size();

arr.insert(someValue); // May be rejected

if (arr.size() == previousSize) {

throw SizeBeyondLimitEx();

}

To know when its time to start a new array.

enne9

But size() returns the number of elements, if I insert a new one it would always be greater than the previous one. Am I missing something?

mrjj

@enne9
Well the logic here is
if the size is the old size after you tried to insert new value its full/failed and you must start new one.

enne9

Thanks, you're right.
In this case, do you know if there is any way to avoid the annoying output QJson: Document too large to store in data structure that I think will pop up once trying to insert after the limit is reached?

mrjj

@enne9
Hi
I dont think so. (besides install message handler and simply remove it)
Else you would have to sum up all data strings to get an estimate of when you have used 128MB.
But im not sure that so clear cut to keep track of.

enne9

Instead, if I have a QJsonObject (already partially filled) and I want to put a QByteArray inside it?
I convert the QByteArray into a QString using QString(bytearray.toBase64());, but I'd like to know if there is a way to understand how to split the QByteArray into partitions to fit inside the QJsonObject that has already some content.

QJsonObject message;
message["header"] = QStringLiteral("test");
message["content"] = QString(bytearray.toBase64());

The problem in the code above is that the QByteArray can be too big and I want to split it into multiple messages.

Do you have any smart ideas about how to handle that? I've checked the documentation but I didn't find any easy method to deal with that.

jsulm

@enne9 Why do you want to squeeze huge binary data into JSON?! JSON is a text format.
Why not store binary data in files and put files names into JSON document?

mrjj

@enne9

Do you have any smart ideas about how to handle that?

Well based on experience, i would really look over the existing code and see how much effort it would be to convert the code to use
https://github.com/nlohmann/json

Why you may ask ?
Well, when if you have to make hacks/workarounds to use a certain technology, at some point
the cost/pain of maintaining/expanding such code often outweighs the cost of writing it again using a technology that removes the need for workarounds.

nlohmann json you would just store what you need. Should handle it easily.

That lib is super flexible so might require almost no code changes.

// create an empty structure (null)
json j;

// add a number that is stored as double (note the implicit conversion of j to an object)
j["pi"] = 3.141;

// add a Boolean that is stored as bool
j["happy"] = true;

// add a string that is stored as std::string
j["name"] = "Niels";

// add another null object by passing nullptr
j["nothing"] = nullptr;

// add an object inside the object
j["answer"]["everything"] = 42;

// add an array that is stored as std::vector (using an initializer list)
j["list"] = { 1, 0, 2 };

// add another object (using an initializer list of pairs)
j["object"] = { {"currency", "USD"}, {"value", 42.99} };

// instead, you could also write (which looks very similar to the JSON above)
json j2 = {
  {"pi", 3.141},
  {"happy", true},
  {"name", "Niels"},
  {"nothing", nullptr},
  {"answer", {
    {"everything", 42}
  }},
  {"list", {1, 0, 2}},
  {"object", {
    {"currency", "USD"},
    {"value", 42.99}
  }}
};