QByteArray to QVariantMap while conserving numerical type

erwan_.-

Hello all,

I am trying to convert the content of an HTTP request to a QVariantMap, perform some actions, and then save the QVariantMap as a Json file.
The QJsonObject allows to do this very easily with the following code snippet:

    QJsonParseError err;
    QJsonDocument body;

    body = QJsonDocument::fromJson(request->getRequest()->collectedData(), &err);
    if (!(err.error == QJsonParseError::NoError)) {
        qInfo(LogCat()) << "Could not decode the body into a Json format. Error: " << err.error;
        return;
    };
    QVariantMap userConfig = body.object().toVariantMap();
 
 //then do some processing and save the Json

My problem is in the numerical conversions. As Json only has a single 'number' type and does not differentiate 'float' or 'int' (and I believe Qt is following the same rule), this is leading to some issues regarding the type of the values.

Take for instance the following Json that we might read from the request:

{
  "pi": 3.14,
  "almost_pi": 3.0 // float
}

This is correctly read in the QByteArray from the request, (i.e. we read almost_pi as 3.0), but when converting it to a QJsonObject, this would be regarded as:

{
  "pi": 3.14,
  "almost_pi": 3 // int
}

Then, if we were to save this as a Json file, this would be saved as presented above.
From a Json aspect, they are exactly the same. However, from my side this is leading to errors in the next steps of my pipeline, because this file will be read by Python scripts that do make the difference between int and float in Jsons.
Is there a way around that that does not require to create our own parser ?
Also, the request I receive could be anything, and does not follow a specific structure, which makes it impossible to use to<Type>() in the QVariantMap.

Thanks

JonB

@erwan_
As you say, JSON has a numeric type and no int vs float. Sorry, but that means you are wrong to be attempting to read the bytes in the text file and distinguish between 3 vs 3.0. Either one is possible from JSON.

because this file will be read by Python scripts that do make the difference between int and float in Jsons.

I don't understand. If you have a JSON file you would only read it with a JSON library. Even from Python. If you read it with pure Python parsing not JSON parsing you are asking for trouble.

I really think you are trying to fit a square peg into a round hole if you pursue this way. If you are using JSON stick with what JSON provides you with.

VRonin

Any chance you can change "almost_pi": 3.0 to "almost_pi": 3.0e+0? that should fix your issue

Axel Spoerl

It's an equation with one too many unknowns: You know it's a number, but you don't know if the type is int or float.
That means, you have to build the variant map programmatically and figure out the type programmatically.
That's actually not a big deal. There are multiple ways do achieve it.

Assumption:
If you know that almost_pi is an int, assign that type if you find the JSON key.
In other words, you rely on the HTTP request to stick to a specific format.
That's usually done by making the format/version part of the JSON content.

Heuristics:
If no specific format is required, just try whether the number fits into an int without deteriorating its value.

This example implements the heuristic approach:

QVariantMap YourClass::toVariantMap(const QJsonObject &obj)
{
    QVariantMap map;
    const QStringList keys = obj.keys();
    for (const auto &key : keys) {
        const QJsonValue &value = obj.value(key);
        switch (value.type()) {
        case QJsonValue::Type::Object:
            // Recurse, just for the fun of it
            map.insert(key, toVariantMap(value.toObject());
            break;
        case QJsonValue::Type::Array:
            // Ooops, this code isn't prepared to handle an array
            break;
        case QJsonValue::Type::Double: {
            const int intVal = value.toInt();
            const QString strVal = value.toString();
            // Test integer fit
            if (QString::number(intVal) == strVal)
                map.insert(key, intVal);
            else
                map.insert(key, value.toDouble());
        }
            break;
        case QJsonValue::Type::Bool:
            map.insert(key, value.toBool());
            break;
        case QJsonValue::Type::String:
            map.insert(key, value.toString());
            break;
        case QJsonValue::Type::Undefined:
            // Handle format error
            break;
        case QJsonValue::Type::Null:
            // Handle or ignore null value
            break;
        }
    }
    return map;
}

JonB

@Axel-Spoerl said in QByteArray to QVariantMap while conserving numerical type:

        const int intVal = value.toInt();
        const QString strVal = value.toString();
        // Test integer fit
        if (QString::number(intVal) == strVal)

But this does not do/does the opposite of what the OP wants. (And you could probably achieve exactly the same more simply without string via conversion via if (value.toInt() != value.toDouble()), with the same problem/flaw.)

The OP wants that he should be to distinguish the following in the JSON input text:

"almost_pi": 3.0 // float
"almost_pi": 3 // int

Using your approach both of these deliver the same "number", because JSON should return (untested) the same QJsonValue::Type::Double value. But their "requirement" is to treat these two differently. Which (so far as I know) can only be done by looking at the actual characters in the JSON input text. (Once JSON has parsed that you just get back a conversion to QJsonValue::Type::Double in all cases.) And that is what the OP wishes to do, which I suggest is not a good idea for a number of reasons. He is not, or should not be, in control of the character output format of the generated JSON, that is up to the JSON library used by the sender and should be allowed to vary as that pleases so long as it is legal JSON.

Axel Spoerl

@JonB
value.toInt() == value.toDouble() is true for "3.0" and "3".
The example would always make "3.0" a double.

That set aside, the solid implementations I have seen solve ambiguous types by format versions or, in complex cases, by additional key/value pairs to specify the format of other keys. It's not entirely clear to me, how the OP intends to interact between Python and C++.
The example was just meant to illustrate a heuristic implementation.

JonB

@Axel-Spoerl said in QByteArray to QVariantMap while conserving numerical type:

The example would always make "3.0" a double.

I really don't understand. May I ask whether you have actually tested you code against two different lines of input:

"almost_pi": 3.0
"almost_pi": 3

You are saying your code distinguishes these two different lines of original text in the JSON are you? I am "surprised", but have not tested. Since I am expecting either 3 or 3.0 in input to deliver identical QJsonValue::Type::Double types and values, but your code relies on them resulting in some difference....

P.S.

by additional key/value pairs to specify the format of other key

That is the only safe and "clean" solution I know of. An alternative, if you really want to receive a different value for a sender floating point versus integer, is to send the values as strings (QJsonValue::Type::String) not numbers (QJsonValue::Type::Double) and do the conversion to int/double yourself accordingly, so that you actually see the incoming string.

Axel Spoerl

Turns out that things are a bit different than I had originally thought, sorry for that.
QJsonValue::toString() returns QString() for both cases:

"almost_pi": 3.0
"almost_pi": 3

There is no way, to get the difference between both, because numeric JSON values can't be retrieved as a string.
That eliminates the heuristic option, and leaves the remaining two: Either interpret almost_pi as a float all the time, or use another key/value pair to specify a type.

Thanks @JonB for spotting!

SimonSchroeder

@erwan_ said in QByteArray to QVariantMap while conserving numerical type:

Then, if we were to save this as a Json file, this would be saved as presented above.

Maybe this is your actual problem: Saving the floating point type from C++ writes out 3 instead of 3.0. In that case you need to specify some formatting to make sure the number is written out in a way that makes it clear it is floating point. Because JSON does not distinguish between integer and floating point numbers there is no reason for QJsonDocument to make that distinction by default.

JonB

@SimonSchroeder said in QByteArray to QVariantMap while conserving numerical type:

In that case you need to specify some formatting to make sure the number is written out in a way that makes it clear it is floating point.

I don't follow you here. How would any kind of formatting of the number at sender side make any difference to the result received at the receiver side using a JSON parser to read it which will return a "numeric"/QJsonValue::Type::Double? Other than sending it is a string type and doing the numeric conversion at receiver side, which is quite different.