How to parse nested array in a Json file

Aymeric_Qt · 7 Jun 2020, 12:51

@mrjj
Yes I'm sure I didn't made any modification, I have the same set of test data in my database from the beginning (and to be sure I've checked directly in the DB).

mrjj · replied to Aymeric_Qt on 7 Jun 2020, 12:51

@Aymeric_Qt

Hmm ok
so it seems to be a differnce between

QJsonDocument::fromJson
and
QJsonDocument allPagesJsonDoc(allPagesArray);

Not a good explanation for that, im afraid.

Christian Ehrlicher · wrote on 7 Jun 2020, 13:04

Can you please provide a minimal example for the behavior? I doubt there is a problem in the Qt api but more in how you convert from/to QString/QByteArray.

Aymeric_Qt · 6 Jul 2020, 15:58

@mrjj, @Christian-Ehrlicher
It might come from the fact that when the file is retrieve is it firstly put inside a QByteArray.

I've tried to qDebug the QByteArray copy and paste the content inside a text editor and then find ² symbol. It seems that the sympbol is converted as xB2.

@Christian-Ehrlicher:
I'm not sure to fully understand your request but I will try to give you the best overview:

First I send a set of test data to my API with Postman here is an example of the request body:

[
	{
		"name" : "Alpha Blob",
		"sign" : "A",
		"count" : 5,
		"rank" : 1,
		"type" : "TYPE_A",
		"tags": [
        	{
            	"id": 1
        	}
    	]
	},
        etc...
]

Then I use my Qt App to get all the datas with the following functions:

This function is called when I click on a button

void Dialog::getAllBlobs()
{
    QJsonDocument allPagesJsonDoc;

    QString url = "blobj/all?pageNumber=";
    if(blobStore.getIsUpToDate())
    {
        allPagesJsonDoc = blobStore.getBlobs();
    }
    else
    {
        allPagesJsonDoc = httpService.getAll(url);
        QString storeStatus = blobStore.storeBlobs(allPagesJsonDoc);
        if(storeStatus != "OK")
        {
            QMessageBox::critical(this, "Error", storeStatus);
        }
    }

    displayResponse(allPagesJsonDoc);
}

On the first call (after the application opened) datas are always fetched through the API.
Here is how I request the API:

QJsonDocument httpService::getAll(QString &url)
{
    qDebug() << "Get all blobs from API";

    int pageNumber = 0;
    bool isLast = false;
    QJsonArray allPagesArray;
    QJsonObject infos;

    do {
        QString requestUrl = url.append(QString::number(pageNumber));
        QJsonDocument objectList = get(requestUrl);

        if(objectList.object()["content"].isNull())
        {
            break;
        }

        allPagesArray.append(objectList.object()["content"]);

        isLast = objectList.object()["last"].toBool();
        pageNumber++;

        if(isLast == true)
        {
            infos.insert("totalPages", objectList.object().value("totalPages"));
            infos.insert("totalElements", objectList.object().value("totalElements"));
            infos.insert("size", objectList.object().value("size"));
            allPagesArray.append(infos);
        }

    } while(!isLast);

    QJsonDocument allPagesJsonDoc(allPagesArray);

    return allPagesJsonDoc;
}

Here is how the GET http request is made:

QJsonDocument httpService::get(QString &url)
{
    QString requestUrl = m_apiUrl +  url;
    QNetworkRequest request(requestUrl);
    request.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");

    QNetworkReply *reply = nam.get(request);

    while (!reply->isFinished())
    {
        qApp->processEvents();
    }

    QByteArray response_data = reply->readAll();

    QJsonDocument json = handleHTTPErrors(response_data, reply);

    reply->deleteLater();

    return json;
}

Then I store the data in a file:

QString blobStore::storeBlobs(QJsonDocument &blobsToSave)
{
    qDebug() << "Storing blobs...";
    QString returnMsg;

    QString blobStore = QDir::currentPath() + QDir::separator() + "blobStore.json";
    QFile file(blobStore);

    if(!file.open(QIODevice::WriteOnly))
    {
        returnMsg = file.errorString();
        return returnMsg;
    }

    QTextStream stream(&file);
    stream << QString(blobsToSave.toJson());
    file.close();

    m_isUpToDate = true;

    returnMsg = "OK";

    return returnMsg;
}

So now my store is considered as up to date. So if I want to retrieve all datas once again, it going to be retrieve from the file.
Here is how:

QJsonDocument blobStore::getBlobs()
{

    qDebug() << "Get all blobs from STORE";
    QJsonParseError jsonError;

    QString blobStore = QDir::currentPath() + QDir::separator() + "blobStore.json";
    QFile file(blobStore);

    if(!file.exists())
    {
        qWarning() << "Blob store file not found";
    }

    if(!file.open(QIODevice::ReadOnly))
    {
        qWarning() << "Cannot open Blob store file: " << file.errorString();
    }

    QByteArray storeContentArray = file.readAll();

    file.close();

    QJsonDocument storeContentJson = QJsonDocument::fromJson(storeContentArray, &jsonError);

    if(jsonError.error != QJsonParseError::NoError)
    {
        qDebug() << QString("JsonError: %1").arg(jsonError.errorString());
        qDebug() << jsonError.offset;
    }
    else if(storeContentJson.isEmpty())
    {
        qDebug() << "storeContentJsonArray is Empty";
    }

    return storeContentJson;
}

Finally datas are dipslaued in a simple Plain Text Editor.

Hope it is what you wanted.

If you want to see all of the code:
https://github.com/AyNeumann/DraftQtProject

Christian Ehrlicher · wrote on 7 Jun 2020, 14:48

Please don't convert the QByteArray to a QString just to pass it to a QTextStream/QFile. Simply use QFile::write(QByteArray()).
And when you read from the file you get a parser error? Can you provide a minimal example (e.g. writing out a simple json array with "²" and read it in again so we can test it? What OS do you use?

Aymeric_Qt · 7 Jun 2020, 18:13

@Christian-Ehrlicher
If I have the following Json:
{
"sign": "²"
}

and try to read it, I have the following error:

"JsonError: invalid UTF8 string"

The QJsonParseError::offset return 13 which is the " character juste before the ² symbol.

I'm on Windows 10.

@Christian-Ehrlicher said in How to parse nested array in a Json file:

Please don't convert the QByteArray to a QString just to pass it to a QTextStream/QFile. Simply use QFile::write(QByteArray()).

I put the response of the get request inside a QJsonDocument because it is for me easier to manipulate to create a consitent object.
I will try to use the raw QByteArray to write into the file but it will take some time as it implies a lot of refactoring.

Christian Ehrlicher · wrote on 7 Jun 2020, 15:59

@Aymeric_Qt said in How to parse nested array in a Json file:

I will try to use the raw QByteArray to write into the file but it will take some time as it implies a lot of refactoring.

Why? It's just a simple change in storeBlobs() which removes two lines and adds one new

Christian Ehrlicher · wrote on 7 Jun 2020, 16:43

And this is a simple reproducer for me:

int main(int argc, char *argv[])
{
    QApplication app(argc, argv);
    
    QJsonArray arr;
    arr += QJsonValue(QString(QChar(0xb2)));
    QFile f(QLatin1String("tmp.txt"));
    f.open(QIODevice::WriteOnly);
    f.write(QJsonDocument(arr).toJson());
    f.close();
    
    f.open(QIODevice::ReadOnly);
    QJsonParseError err;
    QJsonDocument doc(QJsonDocument::fromJson(f.readAll(), &err));
    if (err.error != QJsonParseError::NoError) {
        fprintf(stderr, "Parse error at: %d", err.offset);
        return 1;
    }
    arr = doc.array();
    for (const auto &ele : arr)
        QMessageBox::information(0, QString(), ele.toString());
    return 0;
}

JonB · 6 Jul 2020, 18:13

@Aymeric_Qt said in How to parse nested array in a Json file:

{
 "sign": "²"
}

That's a "funny" character, isn't the error message

"JsonError: invalid UTF8 string"

telling you it's not a UTF-8 character? Seems pretty likely to me.

Aymeric_Qt · 6 Aug 2020, 10:31

@Christian-Ehrlicher

Sorry I did not see your message early enough.

For info I'm a beginner in Qt and in C++ in general so yes, even something which is going to be nothing for you could be really complicated to me.

As you said:
@Christian-Ehrlicher said in How to parse nested array in a Json file:

Please don't convert the QByteArray to a QString just to pass it to a QTextStream/QFile. Simply use QFile::write(QByteArray()).

So I've 'refactor' storeBlob so it can take QByteArray as an argument and save it into a file using:

file.write(blobsToSave);

And in this case it seems to work correctly.

But to be sure I will also try your reproducer.

@JonB :
Yes that what I've thought but it in this case why can it be received from the API, read, displayed and save but not read from a file?

JonB · replied to Aymeric_Qt on 8 Jun 2020, 10:34

@Aymeric_Qt
Because characters have different representations when saved into file.

Aymeric_Qt · wrote on 8 Jun 2020, 16:37

@Christian Ehrlicher,

Ok so I've tested with your reproducer (copy and paste in the main function) and nothing blows up.
So I've tried with the 'storeBlobs' I had and just change the:

QTextStream stream(&file);
stream << QString(blobsToSave.toJson());

into:

file.write(blobsToSave.toJson());

as you did in the code sample you gave me and now it's working I can read data from the file without any problem.

So the problem might came from the fact that I was converting the QJsonDocument into a QTextStream (maybe because I did not did it the right way).

Thank you all for your replies!

Christian Ehrlicher · wrote on 8 Jun 2020, 17:50

@Aymeric_Qt said in How to parse nested array in a Json file:

So the problem might came from the fact that I was converting the QJsonDocument into a QTextStream

It is (and therefore my question about the used OS) - see https://doc.qt.io/qt-5/qtextstream.html#details

"By default, QTextCodec::codecForLocale() is used for reading and writing,"