How to parse nested array in a Json file

Aymeric_Qt

Hello everyone,

I have a problem when I try to create a QJsonDocument which contains nested array.
In fact I get from my API a paged result and I save it into a file. The problem occurs when I want to get the content of the file.
Here is the structure of the Json:

[
    [
        {
          1st object of the first page
        },
        {
          2nd object of the first page
        }
        etc...
    ],
    [
       {
        1st object of the second page
       }
       etc...
    ],
    {
     Object with some informations
    }
]

So the Json is store as it is in a file and it is valid (I've tested it in Visual Studio and Postman and none of them returned any errors).

Here is the code I've written to retrieve it from the file and parse it to create a QJsonDocument:

(File management etc..)
QByteArray storeContentArray = file.readAll(); //not safe with big files

file.close();

QJsonDocument storeContentJson = QJsonDocument::fromJson(storeContentArray, &jsonError);

if(jsonError.error != QJsonParseError::NoError)
{
    qDebug() << QString("JsonError: %1").arg(jsonError.errorString());
}
else if(storeContentJson.isEmpty())
{
    qDebug() << "storeContentJsonArray is Empty";
}

return storeContentJson;

The Json parse error is: "JsonError: invalid UTF8 string" . This code work well with a single Json object or with a 'simple' array but not with nested array.

So is there a way to solve this problem or shall I change the file structure ?

Thanks for reading.

EDIT:
It seems that it is not the nested array that Qt does not like but that would be to have 2 array nested in one array. I've just try to delete the second nested array (the result of the second page) so the Json structure look like this:

[
    [
        {
        1st object of the first page
        },
        etc ...
    ]
]

or like this:

[
    [
        {
        1st object of the first page
        },
        etc ...
    ],
    {
    Some other object
    }
]

And it works fine. So it might be the real problem.

Christian Ehrlicher

And this is a simple reproducer for me:

int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    
    QJsonArray arr;
    arr += QJsonValue(QString(QChar(0xb2)));
    QFile f(QLatin1String("tmp.txt"));
    f.open(QIODevice::WriteOnly);
    f.write(QJsonDocument(arr).toJson());
    f.close();
    
    f.open(QIODevice::ReadOnly);
    QJsonParseError err;
    QJsonDocument doc(QJsonDocument::fromJson(f.readAll(), &err));
    if (err.error != QJsonParseError::NoError) {
        fprintf(stderr, "Parse error at: %d", err.offset);
        return 1;
    }
    arr = doc.array();
    for (const auto &ele : arr)
        QMessageBox::information(0, QString(), ele.toString());
    return 0;
}

mrjj

Hi

JsonError: invalid UTF8 string

Does not sound like it's about the structure
but more about illegal text somewhere.

Also can it not tell you the location in the document of the error ?

Aymeric_Qt

@mrjj said in How to parse nested array in a Json file:

Also can it not tell you the location in the document of the error ?

Hi mrjj ,
I didn't find a way to get this information from Qt.

mrjj

@Aymeric_Qt
Hi
Try to see what
https://doc.qt.io/qt-5/qjsonparseerror.html#offset-var
contains when you get error.

Should give an hint what text it does not like.

Aymeric_Qt

Thanks @mrjj,

I didn't saw that in the documentation and it help me to solve the problem.

Also you were right it is an 'illegal' character that were causing the problem (a ² symbol used as a value).

This is strange because it seems that Qt can receive a Json with this symbol from an API, display it, save it in a file but cannot retrieve it from the file (at least with the QJsonDocument::fromJson method).

So I don't know if I am missing something or if it is a Qt "bug". I will have to find a workaround.

mrjj

@Aymeric_Qt
Hi
When you get it from the APi, do you also parse it ?
I agree that is a bit odd then.

Aymeric_Qt

@mrjj,
Yes because I get the response and put it into a QJsonDocument and then append it into a QJsonArray.
Here is the full code:

QJsonDocument httpService::getAll(QString &url)
{
    int pageNumber = 0;
    bool isLast = false;
    QJsonArray allPagesArray;
    QJsonObject infos;

    do {
        QString requestUrl = url.append(QString::number(pageNumber));
        QJsonDocument objectList = get(requestUrl);

        if(objectList.object()["content"].isNull())
        {
            break;
        }

        allPagesArray.append(objectList.object()["content"]);

        isLast = objectList.object()["last"].toBool();
        pageNumber++;

        if(isLast == true)
        {
            infos.insert("totalPages", objectList.object().value("totalPages"));
            infos.insert("totalElements", objectList.object().value("totalElements"));
            infos.insert("size", objectList.object().value("size"));
            allPagesArray.append(infos);
        }

    } while(!isLast);

    QJsonDocument allPagesJsonDoc(allPagesArray);

    return allPagesJsonDoc;
}

mrjj

@Aymeric_Qt
Ok that is in fact a bit odd.
Are you very sure that invalid text came from the API and not somehow
introduced later on ?
Seems very odd bug if first it can parse and then not :)

Aymeric_Qt

@mrjj
Yes I'm sure I didn't made any modification, I have the same set of test data in my database from the beginning (and to be sure I've checked directly in the DB).

mrjj

@Aymeric_Qt

Hmm ok
so it seems to be a differnce between

QJsonDocument::fromJson
and
QJsonDocument allPagesJsonDoc(allPagesArray);

Not a good explanation for that, im afraid.

Christian Ehrlicher

Can you please provide a minimal example for the behavior? I doubt there is a problem in the Qt api but more in how you convert from/to QString/QByteArray.

Aymeric_Qt

@mrjj, @Christian-Ehrlicher
It might come from the fact that when the file is retrieve is it firstly put inside a QByteArray.

I've tried to qDebug the QByteArray copy and paste the content inside a text editor and then find ² symbol. It seems that the sympbol is converted as xB2.

@Christian-Ehrlicher:
I'm not sure to fully understand your request but I will try to give you the best overview:

First I send a set of test data to my API with Postman here is an example of the request body:

[
	{
		"name" : "Alpha Blob",
		"sign" : "A",
		"count" : 5,
		"rank" : 1,
		"type" : "TYPE_A",
		"tags": [
        	{
            	"id": 1
        	}
    	]
	},
        etc...
]

Then I use my Qt App to get all the datas with the following functions:

This function is called when I click on a button

void Dialog::getAllBlobs()
{
    QJsonDocument allPagesJsonDoc;

    QString url = "blobj/all?pageNumber=";
    if(blobStore.getIsUpToDate())
    {
        allPagesJsonDoc = blobStore.getBlobs();
    }
    else
    {
        allPagesJsonDoc = httpService.getAll(url);
        QString storeStatus = blobStore.storeBlobs(allPagesJsonDoc);
        if(storeStatus != "OK")
        {
            QMessageBox::critical(this, "Error", storeStatus);
        }
    }

    displayResponse(allPagesJsonDoc);
}

On the first call (after the application opened) datas are always fetched through the API.
Here is how I request the API:

QJsonDocument httpService::getAll(QString &url)
{
    qDebug() << "Get all blobs from API";

    int pageNumber = 0;
    bool isLast = false;
    QJsonArray allPagesArray;
    QJsonObject infos;

    do {
        QString requestUrl = url.append(QString::number(pageNumber));
        QJsonDocument objectList = get(requestUrl);

        if(objectList.object()["content"].isNull())
        {
            break;
        }

        allPagesArray.append(objectList.object()["content"]);

        isLast = objectList.object()["last"].toBool();
        pageNumber++;

        if(isLast == true)
        {
            infos.insert("totalPages", objectList.object().value("totalPages"));
            infos.insert("totalElements", objectList.object().value("totalElements"));
            infos.insert("size", objectList.object().value("size"));
            allPagesArray.append(infos);
        }

    } while(!isLast);

    QJsonDocument allPagesJsonDoc(allPagesArray);

    return allPagesJsonDoc;
}

Here is how the GET http request is made:

QJsonDocument httpService::get(QString &url)
{
    QString requestUrl = m_apiUrl +  url;
    QNetworkRequest request(requestUrl);
    request.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");

    QNetworkReply *reply = nam.get(request);

    while (!reply->isFinished())
    {
        qApp->processEvents();
    }

    QByteArray response_data = reply->readAll();

    QJsonDocument json = handleHTTPErrors(response_data, reply);

    reply->deleteLater();

    return json;
}

Then I store the data in a file:

QString blobStore::storeBlobs(QJsonDocument &blobsToSave)
{
    qDebug() << "Storing blobs...";
    QString returnMsg;

    QString blobStore = QDir::currentPath() + QDir::separator() + "blobStore.json";
    QFile file(blobStore);

    if(!file.open(QIODevice::WriteOnly))
    {
        returnMsg = file.errorString();
        return returnMsg;
    }

    QTextStream stream(&file);
    stream << QString(blobsToSave.toJson());
    file.close();

    m_isUpToDate = true;

    returnMsg = "OK";

    return returnMsg;
}

So now my store is considered as up to date. So if I want to retrieve all datas once again, it going to be retrieve from the file.
Here is how:

QJsonDocument blobStore::getBlobs()
{

    qDebug() << "Get all blobs from STORE";
    QJsonParseError jsonError;

    QString blobStore = QDir::currentPath() + QDir::separator() + "blobStore.json";
    QFile file(blobStore);

    if(!file.exists())
    {
        qWarning() << "Blob store file not found";
    }

    if(!file.open(QIODevice::ReadOnly))
    {
        qWarning() << "Cannot open Blob store file: " << file.errorString();
    }

    QByteArray storeContentArray = file.readAll();

    file.close();

    QJsonDocument storeContentJson = QJsonDocument::fromJson(storeContentArray, &jsonError);

    if(jsonError.error != QJsonParseError::NoError)
    {
        qDebug() << QString("JsonError: %1").arg(jsonError.errorString());
        qDebug() << jsonError.offset;
    }
    else if(storeContentJson.isEmpty())
    {
        qDebug() << "storeContentJsonArray is Empty";
    }

    return storeContentJson;
}

Finally datas are dipslaued in a simple Plain Text Editor.

Hope it is what you wanted.

If you want to see all of the code:
https://github.com/AyNeumann/DraftQtProject

Christian Ehrlicher

Please don't convert the QByteArray to a QString just to pass it to a QTextStream/QFile. Simply use QFile::write(QByteArray()).
And when you read from the file you get a parser error? Can you provide a minimal example (e.g. writing out a simple json array with "²" and read it in again so we can test it? What OS do you use?

Aymeric_Qt

@Christian-Ehrlicher
If I have the following Json:
{
"sign": "²"
}

and try to read it, I have the following error:

"JsonError: invalid UTF8 string"

The QJsonParseError::offset return 13 which is the " character juste before the ² symbol.

I'm on Windows 10.

@Christian-Ehrlicher said in How to parse nested array in a Json file:

Please don't convert the QByteArray to a QString just to pass it to a QTextStream/QFile. Simply use QFile::write(QByteArray()).

I put the response of the get request inside a QJsonDocument because it is for me easier to manipulate to create a consitent object.
I will try to use the raw QByteArray to write into the file but it will take some time as it implies a lot of refactoring.

Christian Ehrlicher

@Aymeric_Qt said in How to parse nested array in a Json file:

I will try to use the raw QByteArray to write into the file but it will take some time as it implies a lot of refactoring.

Why? It's just a simple change in storeBlobs() which removes two lines and adds one new

Christian Ehrlicher

And this is a simple reproducer for me:

int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    
    QJsonArray arr;
    arr += QJsonValue(QString(QChar(0xb2)));
    QFile f(QLatin1String("tmp.txt"));
    f.open(QIODevice::WriteOnly);
    f.write(QJsonDocument(arr).toJson());
    f.close();
    
    f.open(QIODevice::ReadOnly);
    QJsonParseError err;
    QJsonDocument doc(QJsonDocument::fromJson(f.readAll(), &err));
    if (err.error != QJsonParseError::NoError) {
        fprintf(stderr, "Parse error at: %d", err.offset);
        return 1;
    }
    arr = doc.array();
    for (const auto &ele : arr)
        QMessageBox::information(0, QString(), ele.toString());
    return 0;
}

JonB

@Aymeric_Qt said in How to parse nested array in a Json file:

{
 "sign": "²"
}

That's a "funny" character, isn't the error message

"JsonError: invalid UTF8 string"

telling you it's not a UTF-8 character? Seems pretty likely to me.

Aymeric_Qt

@Christian-Ehrlicher

Sorry I did not see your message early enough.

For info I'm a beginner in Qt and in C++ in general so yes, even something which is going to be nothing for you could be really complicated to me.

As you said:
@Christian-Ehrlicher said in How to parse nested array in a Json file:

Please don't convert the QByteArray to a QString just to pass it to a QTextStream/QFile. Simply use QFile::write(QByteArray()).

So I've 'refactor' storeBlob so it can take QByteArray as an argument and save it into a file using:

file.write(blobsToSave);

And in this case it seems to work correctly.

But to be sure I will also try your reproducer.

@JonB :
Yes that what I've thought but it in this case why can it be received from the API, read, displayed and save but not read from a file?

JonB

@Aymeric_Qt
Because characters have different representations when saved into file.

Aymeric_Qt

@Christian Ehrlicher,

Ok so I've tested with your reproducer (copy and paste in the main function) and nothing blows up.
So I've tried with the 'storeBlobs' I had and just change the:

QTextStream stream(&file);
stream << QString(blobsToSave.toJson());

into:

file.write(blobsToSave.toJson());

as you did in the code sample you gave me and now it's working I can read data from the file without any problem.

So the problem might came from the fact that I was converting the QJsonDocument into a QTextStream (maybe because I did not did it the right way).

Thank you all for your replies!

Christian Ehrlicher

@Aymeric_Qt said in How to parse nested array in a Json file:

So the problem might came from the fact that I was converting the QJsonDocument into a QTextStream

It is (and therefore my question about the used OS) - see https://doc.qt.io/qt-5/qtextstream.html#details

"By default, QTextCodec::codecForLocale() is used for reading and writing,"

Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

How to parse nested array in a Json file