How to insert entry to json file

[lukutis222]

Hello. In my program I need to be able to read from JSON and also insert a new JSON entry.

My JSON looks like:

{
	"commands":
	[
	{
		"Response type": "STRING",
		"Command name":"test UART0 ping",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping2",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping3",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping4",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping5",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping6",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping7",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},

	{
		"Response type": "STRING",
		"Command name":"Test NVS",
		"Command": "test_nvs",
		"Expected response": "OK",
		"Response retry count":5,
		"Command retry count":1
	}
	]
}

When "INSERT" button is clicked on my application, I fill the structure with the required information and then I must add a new entry to my file.

From what I understand, I need to do the following:

1. "INSERT" button is clicked
2. Read current .json document and find how many lines there are.
3. When I know how many lines are in the document in total, I must return 2 lines and insert a comma ( , )
4. Then starting from the following line, I must insert a whole new entry.

For example, lets say I want to insert:

	{
		"Response type": "STRING",
		"Command name":"NEW INSERTED COMMAND",
		"Command": "new_command",
		"Expected response": "OK",
		"Response retry count":10,
		"Command retry count":2
	}

Then my JSON after inserting would look like:

{
	"commands":
	[
	{
		"Response type": "STRING",
		"Command name":"test UART0 ping",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping2",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping3",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping4",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping5",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping6",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping7",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},

	{
		"Response type": "STRING",
		"Command name":"Test NVS",
		"Command": "test_nvs",
		"Expected response": "OK",
		"Response retry count":5,
		"Command retry count":1
	},
	{
		"Response type": "STRING",
		"Command name":"NEW INSERTED COMMAND",
		"Command": "new_command",
		"Expected response": "OK",
		"Response retry count":10,
		"Command retry count":2
	}
	]
}

I tried to insert a command using the code below:

QString filename="./testplans/filename.json";
    QFile file(filename);
    int line_count=0;
    file.open(QIODevice::ReadWrite | QIODevice::Text);
    QString line[100];
    QTextStream in(&file);
    while( !in.atEnd())
    {
        line[line_count]=in.readLine();
        line_count++;
    }
    qDebug("total line count =%u \n",line_count);
    file.seek(0); //return to the beggining
    file.seek(line_count-2);
    QByteArray test = file.readLine();
    qDebug("read line linecount-2 = %s\n",test.toStdString().c_str());
    test += ",";
    file.write("test");
    file.close();

But it did not seem to work properly, it insert some text to line 8 for some weird reason:

[ChrisW67]

The file seek units are bytes not lines.

For a file as small as the 100 lines you allow for, you should probably use QJsonDocument and related classes.

[lukutis222]

@ChrisW67 Ahh thanks for clarifying regarding the fileseek.

Also, the file size (in lines) can be much bigger than 100. This is just an example

[JonB]

@lukutis222
You are absolutely are not intended to insert anything into a JSON file, with your seeking and line counting and writing into the text file etc. You are supposed to read it in as a JSON file to construct the corresponding object hierarchy, do your manipulations (inserting/deleting/updating) on that, and then resave the whole new hierarchy as a JSON file.

Not to mention: you will not be able to insert anything into a text file by attempting to open it read-write and adding stuff in, that will not work.

[lukutis222]

I updated my code slightly I think I am getting close:

    QString filename="./testplans/filenname.json";
    QFile file(filename);
    int line_count=0;
    file.open(QIODevice::ReadWrite | QIODevice::Text);
    // line is temporary variable to hold all lines information
    QString line[1000]; // can hold up to 1000 lines
    QTextStream in(&file);
    while( !in.atEnd())
    {
        line[line_count]=in.readLine();
        line_count++;
    }
    // after all lines are read, modify the specific line and insert a comma ( , )
    qDebug(" line count-3 = %s \n",line[line_count-3].toStdString().c_str());
    line[line_count-3] += ",";
    qDebug(" line count-3 after inserting comma = %s \n",line[line_count-3].toStdString().c_str());

    //overwrite the whole file with line variable that has been modified.
    QTextStream out(&file);
    out << line << Qt::endl;    
    file.close();

The logic of above code is quite simple. Read current file data line by line and assign it to variable line.

After I read all lines, I modify the required line and insert a comma sign. Then I want to overwrite a whole document with the new updated data.

However, the above code does not seem to work as I expect. Intead of overwriting the whole document and iserting a comma, it inserts some strange text at the bottom of the file:

[lukutis222]

@JonB
I see what you mean. Can you clarify if I understand you correctly:

        QFile file("./testplans/filename.json");
        file.open(QIODevice::ReadWrite | QIODevice::Text);
        QByteArray commands = file.readAll();

        QJsonDocument jsonresponse = QJsonDocument::fromJson(commands);
        QJsonObject jsonObject = jsonresponse.object();
        QJsonArray commandsArray = jsonObject.value("commands").toArray();

Now my commandsArray contain all command entries. I can Insert one more entry and overwrite the whole document. Is that correct?

[jsulm]

@lukutis222 said in How to insert entry to json file:

Is that correct?

Why don't you simply try?

[JonB]

@lukutis222 said in How to insert entry to json file:

Is that correct?

Yes.

[J.Hilk]

@lukutis222 keep in mind, that json does not specify an order of attributes/entries. Simply scanning and counting lines to modify your json document is bound to break very soon

[lukutis222]

@J-Hilk
Yeah I figured. Now I am working directly with json instead of counting lines.. Should be more reliable this way

[lukutis222]

@JonB

I can insert a new entry to my Array, however it is not fully clear to me how can I overwrite the whole file with the inserted entry.

        QFile file("./testplans/filename.json");
        file.open(QIODevice::ReadWrite | QIODevice::Text);
        QByteArray commands = file.readAll();

        QJsonDocument jsonresponse = QJsonDocument::fromJson(commands);
        QJsonObject jsonObject = jsonresponse.object();
        QJsonArray commandsArray = jsonObject.value("commands").toArray();

        // insert new entry
        QJsonObject data_to_insert = QJsonObject(
        {
        qMakePair(QString("Response type"),QJsonValue("test")),
        qMakePair(QString("Command name"), QJsonValue("test")),
        qMakePair(QString("Command"), QJsonValue("test")),
        qMakePair(QString("Expected response"), QJsonValue("test")),
        qMakePair(QString("Response retry count"), QJsonValue("test")),
        qMakePair(QString("Command retry count"), QJsonValue("test"))
        });
        commandsArray.insert(commandsArray.count(),data_to_insert); // insert new entry at the end

      // now my commandsArray consist a new entry

[JonB]

@lukutis222
QByteArray QJsonDocument::toJson(QJsonDocument::JsonFormat format = Indented) const. Save that to file. Do not open it QIODevice::ReadWrite!

[lukutis222]

@JonB

QFile file("./testplans/filename.json");
        file.open(QIODevice::ReadWrite | QIODevice::Text);
        QByteArray commands = file.readAll();

        QJsonDocument jsonresponse = QJsonDocument::fromJson(commands);
        QJsonObject jsonObject = jsonresponse.object();
        QJsonArray commandsArray = jsonObject.value("commands").toArray();

        // insert new entry
        QJsonObject data_to_insert = QJsonObject(
        {
        qMakePair(QString("Response type"),QJsonValue("test")),
        qMakePair(QString("Command name"), QJsonValue("test")),
        qMakePair(QString("Command"), QJsonValue("test")),
        qMakePair(QString("Expected response"), QJsonValue("test")),
        qMakePair(QString("Response retry count"), QJsonValue("test")),
        qMakePair(QString("Command retry count"), QJsonValue("test"))
        });
        commandsArray.insert(commandsArray.count(),data_to_insert); // insert new entry at the end

         // now my commandsArray consist a new entry

         // create new QJsonDocument with updated commandsArray
        QJsonDocument json_doc(commandsArray);
        QByteArray bytearray = json_doc.toJson();
        qDebug() << "json response is " << bytearray;

Is there any particular reason why I cannot see the whole json response when I try to print it to the console using

qDebug() << "json response is " << bytearray;

I see:

json response is  "[\n    {\n        \"Command\": \"ping\",\n        \"Command name\": \"Test UART0 ping\",\n        \"Command retry count\": 1,\n        \"Expected response\": \"pong\",\n        \"Response retry count\": 5,\n        \"Response type\": \"STRING\"\n    },\n    {\n        \"Command\": \"ping\",\n        \"Command name\": \"Test UART0 ping2\",\n        \"Command retry count\": 1,\n        \"Expected response\": \"pong\",\n        \"Response retry count\": 5,\n        \"Response type\": \"STRING\"\n    },\n    {\n        \"Command\": \"ping\",\n        \"Command name\": \"Test UART0 ping3\",\n        \"Command retry count\": 1,\n        \"Expected response\": \"pong\",\n        \"Response retry count\": 5,\n        \"Response type\": \"STRING\"\n    },\n    {\n        \"Command\": \"ping\",\n        \"Command name\": \"Test UART0 ping4\",\n        \"Command retry count\": 1,\n        \"Expected response\": \"pong\",\n        \"Response retry count\": 5,\n        \"Response type\": \"STRING\"\n    "��

The last command printed out is "Test UART0 ping4" which is definately not the last one.

Also, what is the problem of opening in ReadWrite, since I am reading the file and also will be overwriting it.

[JonB]

@lukutis222
Maybe qDebug() << QByteArray only prints a certain maximum of characters, I don't know. Maybe qDebug() << QString(bytearray) would work better, I don't know. In any case relying on the output of qDebug() is not a good test.

Also, what is the problem of opening in ReadWrite, since I am reading the file and also will be overwriting it.

You are not reading and writing/overwriting the file at the same time. You are/should be:

• Open file for read.
• Read whole file.
• Close file.
• Change in-memory stuff.
• Open file for write/overwrite (i.e. with truncate).
• Write whole file.
• Close file.

Do not do seeks and writes on a read-write file. You will get away with it if the file happens to be longer than before, but you will be left with incorrect extra stuff if it happens to be shorter.

[mpergand]

@lukutis222 said in How to insert entry to json file:

qDebug() << "json response is " << bytearray;

For easier reading use:
qDebug().noquote()<<bytearray;

[lukutis222]

I am almost there! (I have fixed the issue with qDebug not printing out the whole text and it turned out to be a syntax error inside json).

My current code

        QFile file("./testplans/filename.json");
        file.open(QIODevice::ReadOnly | QIODevice::Text);
        QByteArray commands = file.readAll();

        QJsonDocument jsonresponse = QJsonDocument::fromJson(commands);
        QJsonObject jsonObject = jsonresponse.object();
        QJsonArray commandsArray = jsonObject.value("commands").toArray();

        // insert new entry
        QJsonObject data_to_insert = QJsonObject(
        {
        qMakePair(QString("Response type"),QJsonValue("test")),
        qMakePair(QString("Command name"), QJsonValue("test")),
        qMakePair(QString("Command"), QJsonValue("test")),
        qMakePair(QString("Expected response"), QJsonValue("test")),
        qMakePair(QString("Response retry count"), QJsonValue("test")),
        qMakePair(QString("Command retry count"), QJsonValue("test"))
        });
        commandsArray.insert(commandsArray.count(),data_to_insert); // insert new entry at the end

         // now my commandsArray consist a new entry

         // create new QJsonDocument with updated commandsArray
        QJsonDocument json_doc(commandsArray);
        QByteArray bytearray = json_doc.toJson();
        qDebug() << "json response is " << bytearray;


        file.open(QFile::WriteOnly|QFile::Truncate);
        file.write(bytearray);
        file.close();

Everything works as expected, just one issue:

Initially, my json looks like (I have shorted it up just to make easier to work with):

{
	"commands":
	[
	{
		"Response type": "STRING",
		"Command name":"Test UART0 ping",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping2",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	},
	
		{
		"Response type": "STRING",
		"Command name":"Test UART0 ping3",
		"Command": "ping",
		"Expected response": "pong",
		"Response retry count":5,
		"Command retry count":1
	}
	]
}

After overwriting:

[
    {
        "Command": "ping",
        "Command name": "Test UART0 ping",
        "Command retry count": 1,
        "Expected response": "pong",
        "Response retry count": 5,
        "Response type": "STRING"
    },
    {
        "Command": "ping",
        "Command name": "Test UART0 ping2",
        "Command retry count": 1,
        "Expected response": "pong",
        "Response retry count": 5,
        "Response type": "STRING"
    },
    {
        "Command": "ping",
        "Command name": "Test UART0 ping3",
        "Command retry count": 1,
        "Expected response": "pong",
        "Response retry count": 5,
        "Response type": "STRING"
    },
    {
        "Command": "test",
        "Command name": "test",
        "Command retry count": 2,
        "Expected response": "test",
        "Response retry count": 1,
        "Response type": "test"
    }
]

As you can see from above, there is an issue with:

        QJsonDocument json_doc(commandsArray);
        QByteArray bytearray = json_doc.toJson();

I need to make sure that my commandsArray is encapsulated inside an array named "commands". I am now looking into how I can achieve this.

Also, the order is all messed up but I have read that everyone have this issue.

[JonB]

@lukutis222
You cannot alter the order/formatting of the toJson() output, JSON data is unordered, what it produces is legal JSON, may not be in same order/format as the original file.

You created a complete QJsonDocument via QJsonDocument json_doc(commandsArray). That is not what it originally was, you have lost what came before it/where the array was in the document.

I can't remember the details of how QJsonDocument works. Hopefully the changes you made went into the QJsonDocument jsonresponse directly. Did you try jsonresponse.toJson() to verify that (after your updates), no creation of a new QJsonDocument json_doc?

[lukutis222]

@JonB

Thanks for clarifying regarding the order.

Yes I now realize that I created QJsonDocument using commandsArray that I modified and it does not contain the main JsonObject "commands" . I have try to check if the change I made to commandsArray affect jsonresponse but unfortunately it does not.

I looked up how to add jsonarray inside another json object here (for java language):
https://stackoverflow.com/questions/12142238/add-jsonarray-to-jsonobject

and they use put() method, so something like:

QJsonObject jsonobject;
jsonobject.put("commands", commandsArray);

Unfortunately, QT does not have this method and I was not able to find a relevant method for adding a QJsonArray inside another QJsonObject.

[jsulm]

@lukutis222 said in How to insert entry to json file:

QT does not have this method and I was not able to find a relevant method for adding a QJsonArray inside another QJsonObject.

https://doc.qt.io/qt-5/qjsonobject.html#insert

[lukutis222]

@jsulm
Yes you are right.

        QJsonObject new_object;
        new_object.insert("commands",commandsArray);
        QJsonDocument json_doc(new_object);
        QByteArray bytearray = json_doc.toJson();
        file.open(QFile::WriteOnly|QFile::Truncate);
        file.write(bytearray);
        file.close();

Works without any issues. The file now contains everything:

{
    "commands": [
        {
            "Command": "ping",
            "Command name": "Test UART0 ping",
            "Command retry count": 1,
            "Expected response": "pong",
            "Response retry count": 5,
            "Response type": "STRING"
        },
        {
            "Command": "ping",
            "Command name": "Test UART0 ping2",
            "Command retry count": 1,
            "Expected response": "pong",
            "Response retry count": 5,
            "Response type": "STRING"
        },
        {
            "Command": "ping",
            "Command name": "Test UART0 ping3",
            "Command retry count": 1,
            "Expected response": "pong",
            "Response retry count": 5,
            "Response type": "STRING"
        },
        {
            "Command": "test",
            "Command name": "test",
            "Command retry count": 2,
            "Expected response": "test",
            "Response retry count": 1,
            "Response type": "test"
        }
    ]
}