Parsing incomplete json array on QTcpSocket readyRead() , unterminated string

[rifky]

anyone can help me? i cant parsing incomplate jsonarray

json output on debug log

[ { "cellNumber": 3, "uptime": 1598421564 }, { "cellNumber": 6, "up
time": 1598421754 } ]

it should be like this

QJsonDocument([{"cellNumber":3,"uptime":1598421564},{"cellNumber":6,"uptime":1598421754}])

here is my snippet code

QByteArray data = tcpSocketUpdata->readAll();

void MainWindow::getDataForTable(QByteArray data)
{
    if(data.size() > 0){

        QJsonParseError parseError;
        QJsonDocument document = QJsonDocument::fromJson(data, &parseError);

        if (parseError.error == QJsonParseError::NoError){

            if (document.isArray())
            {
                qDebug() << "Document contain array";

                //assign result scanner ke array
                QJsonArray arr_result_scan;
                QJsonObject obj = document.object();

                arr_result_scan = document.array();

                QJsonDocument newDocs;
                newDocs.setArray(arr_result_scan);
//                qDebug()<<"new docs"<<newDocs;

                QTimer *timer = new QTimer();
                timer->setSingleShot(true);
                timer->start();
                QTimer::singleShot(500, this, std::bind(&MainWindow::tambahDataTableView, this,newDocs));

            }
        }else{
            qDebug() << "Parse error: " << parseError.errorString();
        }
    }
}

[jsulm]

@rifky said in Parsing incomplete json array on QTcpSocket readyRead() , unterminated string:

QByteArray data = tcpSocketUpdata->readAll();

Where is this executed? My guess is that you call it in a slot connected to readyRead() signal. You have to consider that you can't assume that if readyRead() signal is emitted you get the whole package when calling readAll(). You have to accumulate the data until you got everything and then parse JSON.

[mrjj]

@rifky said in [Parsing incomplete json array on QTcpSocket readyRead() , unterminated string]

QByteArray data = tcpSocketUpdata->readAll();

Hi
Have you checked that you get all data?

Despise the name readAll it might not read all data
if it comes in more than one packet.

So my guess is that it comes as multiple reads.

update: @jsulm also had that feeling :)

[rifky]

@jsulm i made custom instance QTcpSocket in order to handle mutiple connection,
forked original source

/*------SocksUpdata class-----*/
SocksUpdata::SocksUpdata(QTcpSocket* socket)
{
    tcpSocketUpdata = socket;
    connect (tcpSocketUpdata, &QTcpSocket::readyRead, this, &SocksUpdata::slotReadyRead);
    connect (tcpSocketUpdata, &QTcpSocket::disconnected, this, &SocksUpdata::slotDisconnected);
    connect (tcpSocketUpdata, &QTcpSocket::connected, this, &SocksUpdata::slotConnected);
}
QByteArray SocksUpdata::getData()
{
    QByteArray ba = tcpSocketUpdata->readAll();
    return ba;
}

/*------UpdataServer class-----*/
void UpdataServer::startListen()
{
    server = new QTcpServer();
    connect(server, &QTcpServer::newConnection,this, &UpdataServer::slotNewConnection);
    if(!server->listen(QHostAddress::Any, UPDATA_PORT)){
        qDebug()<< "updataServer Could Not be Started port "<<UPDATA_PORT;
        return;
    }else{
        qDebug()<< "updataServer Started port "<<UPDATA_PORT;
    }
}

void UpdataServer::slotNewConnection()
{
    QTcpSocket* socket = server->nextPendingConnection();
    SocksUpdata* handle_mutipleSocks = new SocksUpdata(socket);

    //setSocketOption
    socket->setSocketOption(QAbstractSocket::KeepAliveOption, 1);
    socket->setSocketOption(QAbstractSocket::LowDelayOption, 0);


    qDebug()<<handle_mutipleSocks->getPerrAddress()<<" Connected to updataserver";

    //! hs1
    socket->write("Welecome");

    //! pairing peeraddress with socketptr
    handle_mutipleSocks->paired_device.insert(socket->peerAddress(),socket);

    connect(handle_mutipleSocks, &SocksUpdata::dataReady,this, &UpdataServer::slotReceive);
    connect(handle_mutipleSocks, &SocksUpdata::socketDisconnected,this, &UpdataServer::slotDisconnectSocket);
}

void UpdataServer::slotReceive(SocksUpdata* socket)
{
    QByteArray dataReceived = socket->getData();
 qDebug().noquote().noquote()<<" UpdataServer::slotReceive from "<<socket->getPerrAddress()<<" "<<dataReceived;
}

how i can handle it if socket receive complete json with separate payload?

i got similar issue but, it's different language i used
https://stackoverflow.com/questions/53937434/parsing-incomplete-json-array

[JonB]

@rifky

QByteArray SocksUpdata::getData()
{
    QByteArray ba = tcpSocketUpdata->readAll();
    return ba;
}
...
void UpdataServer::slotReceive(SocksUpdata* socket)
{
    QByteArray dataReceived = socket->getData();
 qDebug().noquote().noquote()<<" UpdataServer::slotReceive from "<<socket->getPerrAddress()<<" "<<dataReceived;
}

As both @jsulm & @mrjj have said, this does not guarantee you have received all the data sent from the client. At best it guarantees you have received the first byte of whatever may have been sent!

@jsulm correctly stated:

You have to accumulate the data until you got everything and then parse JSON.

We don't know what protocol you are using between your client & server. It is your job to "invent" one. For example, the client might precede its messages with a byte count, or you might have it send a unique \0 byte to indicate when it has finished sending the document. Then your receiver side must keep going readAll() in its readyRead() socket, buffering up till the protocol indicates it has received the full transfer before trying to parse it.

You may want to look at QDataStream transactions and the example https://doc.qt.io/qt-5/qtnetwork-fortuneclient-example.html

[rifky]

@mrjj said in Parsing incomplete json array on QTcpSocket readyRead() , unterminated string:

@rifky said in [Parsing incomplete json array on QTcpSocket readyRead() , unterminated string]

QByteArray data = tcpSocketUpdata->readAll();

Hi
Have you checked that you get all data?

Despise the name readAll it might not read all data
if it comes in more than one packet.

So my guess is that it comes as multiple reads.

update: @jsulm also had that feeling :)

I haven't done it

[KroMignon]

@rifky said in Parsing incomplete json array on QTcpSocket readyRead() , unterminated string:

I haven't done it

As @jsulm and @mrjj have already written, TCP is a stream protocol. When you are sending data from one endpoint to counterpart, all data will arrives but not necessary at the same time. So you can have for QTcpSocket::write() one or more QTcpSocket::dataReady() events.
You have to store data on receiver side until "message is completely received and then process the full message.

There are many strategies to achieve this depending on the way you are using the TCP Socket.

[rifky]

@KroMignon said in Parsing incomplete json array on QTcpSocket readyRead() , unterminated string:

As @jsulm and @mrjj have already written, TCP is a stream protocol. When you are sending data from one endpoint to counterpart, all data will arrives but not necessary at the same time. So you can have for QTcpSocket::write() one or more QTcpSocket::dataReady() events.
You have to store data on receiver side until "message is completely received and then process the full message.
There are many strategies to achieve this depending on the way you are using the TCP Socket.

okay, i'm confused sometimes the issue appears,

what if
on readyRead() function :

readAll response
parsing to jsonarray
if error parsing
assign response to temp_variabel
finding close bracket json until found
if found 
combine with temp_variabel
then parsing to jsonarray

I thought about it like that shown here

i have very little knowledge about qt, can you help me with that ?

thanks

[jsulm]

@rifky said in Parsing incomplete json array on QTcpSocket readyRead() , unterminated string:

assign response to temp_variabel

be careful where you define this variable!
It has to be a class member and not local variable!

[rifky]

solved , maybe it's not a great solution, but it solves my problem

here is my snippet

QJsonParseError parseError_2;
    QJsonParseError parseError_3;
    QByteArray temp_arr;
    QJsonDocument document_2 = QJsonDocument::fromJson(byte_json.data(), &parseError_2);

    if(parseError_2.error != QJsonParseError::NoError){
        emit setJsonInvalid(this->epoch, byte_json);

        for (int k=0;k <= this->epoch; k++) {
            temp_arr.push_back(strJsonInvalid[k]);
        }
        //coba parsing
        QJsonDocument coba_parsing = QJsonDocument::fromJson(temp_arr, &parseError_3);

        // jika parsing sukses , clear object strJsonInvalid
        // reset epoch jadi 0
        if(parseError_3.error == QJsonParseError::NoError){
            set_JsonValid(coba_parsing);//fire signal for sending valid json
            strJsonInvalid.clear();
        }else{
            //jika masih gagal parsing increment epoch (+1)
            setEpoch(this->epoch);
            this->epoch++;
        }
    }

[KroMignon]

@rifky said in Parsing incomplete json array on QTcpSocket readyRead() , unterminated string:

solved , maybe it's not a great solution, but it solves my problem

As @jsulm has written, the best way is to declare a class member to store temporary value.
Then, according to your example, you are search a string with is between [].
So one solution would be something like this:

class SocksUpdata : public QObject
{
    Q_OBJECT

private:
    QByteArray _buffer;
...
};


QByteArray SocksUpdata::getData()
{
    _buffer.append(tcpSocketUpdata->readAll());

    int idx = _buffer.indexOf('[');
    if(_buffer[0] != '[' && idx > 0)
    {
       _buffer.remove(0, idx);
    }
    idx = _buffer.indexOf(']');
    if(_buffer[0] == '['  &&  idx > 0)
    {
        QByteArray byte_json = _buffer.left(idx+1);
        _buffer.remove(0, idx+1);
        QJsonParseError parseErr;
        QJsonDocument doc =  QJsonDocument::fromJson(byte_json, &parseErr);
         if(parseError_2.error != QJsonParseError::NoError)
         {
             emit setJsonInvalid(this->epoch, byte_json);
         }
         else
         {
             // do your stuff
         }
    }
}

[rifky]

@KroMignon said in Parsing incomplete json array on QTcpSocket readyRead() , unterminated string:

@rifky said in Parsing incomplete json array on QTcpSocket readyRead() , unterminated string:

solved , maybe it's not a great solution, but it solves my problem

As @jsulm has written, the best way is to declare a class member to store temporary value.
Then, according to your example, you are search a string with is between [].
So one solution would be something like this:

class SocksUpdata : public QObject
{
    Q_OBJECT

private:
    QByteArray _buffer;
...
};


QByteArray SocksUpdata::getData()
{
    _buffer.append(tcpSocketUpdata->readAll());

    int idx = _buffer.indexOf('[');
    if(_buffer[0] != '[' && idx > 0)
    {
       _buffer.remove(0, idx);
    }
    idx = _buffer.indexOf(']');
    if(_buffer[0] == '['  &&  idx > 0)
    {
        QByteArray byte_json = _buffer.left(idx+1);
        _buffer.remove(0, idx+1);
        QJsonParseError parseErr;
        QJsonDocument doc =  QJsonDocument::fromJson(byte_json, &parseErr);
         if(parseError_2.error != QJsonParseError::NoError)
         {
             emit setJsonInvalid(this->epoch, byte_json);
         }
         else
         {
             // do your stuff
         }
    }
}

thx alot sir, it's working .

i got little bit confuse using QbyteArray, i'm trying to made my code working with QHash wich can storing QByteArray, thx for help i'm realy appriciate that :)