QJsonDocument loses accuracy on long long (int64) data

rushmore

So I'm reading this json data online and try to parse it by first converting it to QVariantMap, by this very convenient call:

QJsonDocument doc = QJsonDocument::fromJson(rawdata);
QVariantMap map = doc.toVariant().toMap();

However, I found some really strange behavior. In my data, there is long integer like this:

"id" : 17847032503102456

QJson automatically convert it to double, but often loses accuracy. So when I turn it to LongLong or QString, it will become something like this: "17847032503102457" or something else close to it (seems very random). But I need to make it exactly like it was passed to me. How can I convert such long integer as string without losing accuracy? This is killing me :-(

Thank you!

kshegunov

@rushmore said:

How can I convert such long integer as string without losing accuracy?

Require the site/service that sends you the JSON to actually send it as a string.

'id': '17847032503102456'

should work just fine!

rushmore

@kshegunov Sorry but this is not possible. They will not change their API because of one guy's request. So there is no good solution for this?

Paul Colby

Hi @rushmore,

What version of Qt are you using?

Using JSON numbers as IDs is bad, because the JSON spec places no requirements on the level of precision a parser must support. So IDs really should be strings, not numbers, but I understand that often data is outside of your control.

It might be small consolation in this case, but I can get back the original ID using:

QString("%1").arg(map.value("id").toDouble(),0,'f',0)

eg:

    QJsonDocument doc1 = QJsonDocument::fromJson("{ \"id\" : 17847032503102456 }");
    QVariantMap map = doc1.toVariant().toMap();
    qDebug() << doc1;
    qDebug() << map;
    qDebug() << doc1.object().value("id").toDouble();
    qDebug() << map.value("id").toString();
    qDebug() << QString("%1").arg(doc1.object().value("id").toDouble(),0,'f',0);
    qDebug() << QString("%1").arg(map.value("id").toDouble(),0,'f',0);
    qDebug() << qVersion();

Outputs:

QJsonDocument({"id":17847032503102456})
QMap(("id", QVariant(double, 1.7847e+16) ) ) 
1.7847e+16
"1.78470325031025e+16"
"17847032503102456"
"17847032503102456"
5.4.2

Good luck.

rushmore

@Paul-Colby said:

QJsonDocument doc1 = QJsonDocument::fromJson("{ "id" : 17847032503102456 }");
QVariantMap map = doc1.toVariant().toMap();
qDebug() << doc1;
qDebug() << map;
qDebug() << doc1.object().value("id").toDouble();
qDebug() << map.value("id").toString();
qDebug() << QString("%1").arg(doc1.object().value("id").toDouble(),0,'f',0);
qDebug() << QString("%1").arg(map.value("id").toDouble(),0,'f',0);

Thanks for the answer. I tried both Qt 5.5.1 and 5.6.0. The case I gave you actually did work. But if you try this: 17846740159128499, it will fail. It's kind of random, which is really frustrating.

On another note, the debug info printed-out indicating the error occurs at QJsonDocument::fromJson(), so it has nothing to do with converting it to QVariantMap. So I changed the title of this thread a little bit.

kshegunov

@rushmore
So here's the core of the problem:
The mantissa of the double is limited and provides a limited number of significant digits (about 15 decimal digits) and floating point numbers are kept normalized, so your number is kept at the boundary of the double's ability to accurately represent. While the floating points can represent a huge dynamic range, this doesn't mean the absolute distances between represented numbers are constant (when the exponent grows numbers get further apart).

QJson automatically convert it to double, but often loses accuracy.

I'm not intimate with how QJsonDocument keeps data internally, but my assumption is that it shouldn't make conversions implicitly. Your number does, however, fit in a 64 bit integer so converting it to such type shouldn't present a problem. Whether or not long long int is indeed 64 bits in size is sadly an implementation detail, C++ requires only that it shouldn't be smaller than long int, which in turn shouldn't be smaller than int. So the data type size will depend on the compiler's implementation. What compiler are you using?

So there is no good solution for this?

I can't think of a good portable way of achieving this, sorry. I can only suggest to look up Qt's source and check whether any conversions are done internally when QJsonDocument is instantiated (parsed from a string). If they are, little you can do about it, beside patching up Qt's source. If not, then converting the variants to a long long should work, if long long is 64 bits in size (you can check that either in the compiler documentation or by using sizeof() with the aforementioned type).

Kind regards.

rushmore

The only solution I can think of is to convert long long numbers to strings before parsing as json. Given the data I get is quite big and have lots of similar data type, that would be painful and performance lowering. :-(

kshegunov

@rushmore

I got curious and checked how the parsing is done. QJsonDocument uses a simple top-down parser, and indeed a conversion to double is done when the number doesn't fit in a regular int. Then the actual conversion is implemented, which is strange as the standard library provides such a function, and judging by the comment in the json parser it doesn't handle loss of precision (only speculating, I haven't checked the actual code closely).

So, if I'm interpreting the code correctly, I do think you should file a bug report for this.

JKSH

This isn't just a problem for QJsonDocument, but with JSON parsers and encoders everywhere: https://cdivilly.wordpress.com/2012/04/11/json-javascript-large-64-bit-integers/

The JSON standard says that we should use the format of JavaScript numbers. The JavaScript standard says that all numbers should be stored as double-precision floating points... but this format doesn't have enough precision to store 64-bit integers.

In an ideal world, the the JSON (or JavaScript) standard would be updated to support higher-precision numbers.

@kshegunov said:

So, if I'm interpreting the code correctly, I do think you should file a bug report for this.

This is not a bug per se, as QJsonDocument is simply following the standard.

Pragmatically though, this would be a very useful feature. We can submit a feature request for QJsonDocument to read/write int64, but this should be a non-default option because it is not standards-compliant. Here's an existing report: https://bugreports.qt.io/browse/QTBUG-28560

While we're at it, here are some other useful "numbers" that JSON doesn't support:

+Inf/-Inf
NaN

kshegunov

@JKSH said:

This is not a bug per se, as QJsonDocument is simply following the standard.
Pragmatically though, this would be a very useful feature.

Possibly, but if I were implementing the parser originally I would have added it, even if it doesn't strictly follow the standard. Especially since it wouldn't actually complicate the parsing.

While we're at it, here are some other useful "numbers" that JSON doesn't support:

If I recall correctly what I saw in the source, though, it seems that Qt actually handles these gracefully ... :)