QJsonObject, how to get size in terms of bytes

SPlatten

I have a strange problem which I'm trying to resolve, I am transmitting JSON over UDP and now have a problem where an error occurs:

Datagram was too large to send

I have built up an instance of QJsonObject and added a binary data block to it, before adding the binary block to the object the size of the object is 167 bytes. After adding it jumps to 147476. Something is obviously wrong, this is my attempt to debug it, but I cannot see where the huge increase is coming from:

QByteArray baBLOB(varValue.toByteArray());
std::string stdBLOB(baBLOB.constData(), baBLOB.length());
QString strBLOB(QString::fromStdString(stdBLOB));
QJsonValue valBLOB(strBLOB);

qDebug() << __FILE__ << __LINE__ << " baBLOB Length: " << baBLOB.length();
qDebug() << __FILE__ << __LINE__ << " stdBLOB Length: " << stdBLOB.length();
qDebug() << __FILE__ << __LINE__ << " strBLOB Length: " << strBLOB.length();

QString strJSON(SckServer::jsonToString(objResponse));
qDebug() << __FILE__ << __LINE__ << " objResponse Length: " << strJSON.length();
objResponse.insert(SckServer::mscszChunk, valBLOB);

strJSON = SckServer::jsonToString(objResponse);
qDebug() << __FILE__ << __LINE__ << " objResponse Length: " << strJSON.length();

And the results:

..\src\datasetmsg.cpp152 baBLOB Length: 32640
..\src\datasetmsg.cpp153 stdBLOB Length: 32640
..\src\datasetmsg.cpp154 strBLOB Length: 32555        Why is this less than the stdBLOB length ?
..\src\datasetmsg.cpp156 objResponse Length: 167      This is the correct length before adding the BLOB
..\src\datasetmsg.cpp159 objResponse Length: 147476   Why is this not just the length of the label and the BLOB?

I understand the first two lines of the output, where the length of the byte array is the same as the std string, but why does the QString version of the data shrink?

The first response 167 shows the object before the payload Chunk of binary data is added to it, when I say binary I am expecting to see a hex string of each nibble.

Why does the JSON value version of the data end up at 147476 and how can I fix it so I have the data the way I want it which is as a hex string of nibbles in the Chunk.

I would expect to see the Chunk at a length of 2 * 32640 = 65280 + the original JSON size of 167 so a total size of 65447, which is under the UDP max size.

Reading https://doc.qt.io/qt-5/qstring.html#fromStdString the function ** fromStdString** returns a string converted to Unicode.

JKSH

@SPlatten said in QJsonObject, how to get size in terms of bytes:

when I say binary I am expecting to see a hex string of each nibble.

Then you must call QByteArray::toHex()

VRonin

@SPlatten said in QJsonObject, how to get size in terms of bytes:

QByteArray baBLOB(varValue.toByteArray());
std::string stdBLOB(baBLOB.constData(), baBLOB.length());
QString strBLOB(QString::fromStdString(stdBLOB));
QJsonValue valBLOB(strBLOB);

What is this?! basically you are in the wild west of string encodings (I actually can't be asked to track all the encoding conversions going on here)

If baBLOB is binary data and not a string, the usual way to hadle this is:

objResponse.insert(SckServer::mscszChunk,QLatin1String(baBLOB.toBase64()));

SPlatten

@VRonin , yes baBLOB is a byte array containing a BLOB from the database.

JKSH

@SPlatten said in QJsonObject, how to get size in terms of bytes:

when I say binary I am expecting to see a hex string of each nibble.

Then you must call QByteArray::toHex()

ChrisW67

@SPlatten said in QJsonObject, how to get size in terms of bytes:

QString strBLOB(QString::fromStdString(stdBLOB));

This with take the binary data in stdBLOB and, treating it as UTF-8 encoded text, attemp to produce a matching QString. Very rarely will an arbitrary binary blob consist entirely of valid UTF-8 data, so the result is almost certainly differ by the number of byte combinations that were invalid UTF-8.

AxelVienna

@SPlatten
What does your socket server do? Is it the database? I am asking because the size explosion happens after you call the server. A couple of bytes here and there during Wild West string conversions may happen (and corrupt your data). The socket server, however, is the Blackbox that seems responsible for the massive increase.

SPlatten

@AxelVienna I think the size growing is explained by the UTF8 encoding from the raw data in the byte array.

SPlatten

@VRonin , here is the modified code using QLatin1String and the output, still not what I expected I the final output:

QByteArray baBLOB(varValue.toByteArray());
QLatin1String latStr(QLatin1String(baBLOB.toBase64()));
std::string stdBLOB(baBLOB.constData(), baBLOB.length());
QString strBLOB(QString::fromStdString(stdBLOB));
qDebug() << __FILE__ << __LINE__ << " baBLOB Length: " << baBLOB.length();
qDebug() << __FILE__ << __LINE__ << " latStr.length: " << latStr.size();
qDebug() << __FILE__ << __LINE__ << " stdBLOB Length: " << stdBLOB.length();
qDebug() << __FILE__ << __LINE__ << " strBLOB Length: " << strBLOB.length();
QString strJSON(SckServer::jsonToString(objResponse));
qDebug() << __FILE__ << __LINE__ << " objResponse Length: " << strJSON.length();
objResponse.insert(SckServer::mscszChunk, latStr);
strJSON = SckServer::jsonToString(objResponse);
qDebug() << __FILE__ << __LINE__ << " objResponse Length: " << strJSON.length();

Output:

..\src\datasetmsg.cpp154 baBLOB Length: 32640
..\src\datasetmsg.cpp155 latStr.length: 43520
..\src\datasetmsg.cpp156 stdBLOB Length: 32640 Not used left in just for illustration
..\src\datasetmsg.cpp157 strBLOB Length: 32555 Not used left in just for illustration
..\src\datasetmsg.cpp159 objResponse Length: 167 Before adding latStr
..\src\datasetmsg.cpp162 objResponse Length: 203333 After adding latStr, again, why the huge increase?

SPlatten

@JKSH , thank you, this is exactly what I wanted.