QTCP and packet parsing

[Jonnybravo]

I am working on a server that is simply recvieing traffic from a client. I am using QT5.9.2 and MSVS 2017 64bit complier to recv the data and parse the data and read it and send it to the client.

So this is my code so far..

void Server::onReadyRead(QObject *socketObject)
{
    QTcpSocket *socket = qobject_cast<QTcpSocket *>(socketObject);
    if (!socket || !socket->bytesAvailable())
    {
        qDebug() << "socket bytes unavailable";
        return;
    }
    qDebug() << "recvd signal data. bytesAvailable:" << socket->bytesAvailable();

    // Recvd the entire buffer from bytesAvailable()
    recvdbuffer = socket->readAll();

    if(recvdbuffer.isEmpty())
    {
		qDebug() << "recvdbuffer has no bytes available";
        return;
    }

    qDebug() << "recvbuffer in hex and size:" << recvdbuffer.toHex() << recvdbuffer.size();

    if(recvdbuffer.length() > MAX_BUFFER )
    {
        socket->close();
        return;
    }
    recdPacket(recvbuffer.toHex(0), recvbuffer[1], recvbuffer.size());
}

so i have a packet that looks like this...

ce06000038000000010000000000000068656c6c6f616c6c00

How would i be able to split this data up into pieces. The recvbuffer.toHex(0) prints it out to hex do i need to convert it?

how to convert hex to string?

[Taz742]

@Jonnybravo said in QTCP and packet parsing:

how to convert hex to string?

QByteArray hex =  QByteArray::fromHex("ce06000038000000010000000000000068656c6c6f616c6c00");
QString str2 = QString::fromUtf16((char16_t*)hex.data());

[Jonnybravo]

@Taz742 It doesn't do what you are expecting.

QByteArray hex =  QByteArray::fromHex("ce06000038000000010000000000000068656c6c6f616c6c00");
QString str2 = QString::fromUtf16((char16_t*)hex.data());
qDebug() << str2;

this returns "?"

instead of "helloall" the string inside the hex..

only thing i can think of is looping thru the array base on a postion and return the buffer..

[SGaist]

Hi and welcome to devnet,

In what form are you supposed to receive your data ?
Is there any protocol between your devices ?

[Jonnybravo]

@SGaist Thankyou. Its just a normal tcp connection the recvdbuffer = socket->readAll(); returns the entire packet in full i'm now just trying to extract the data from the array.

ce
06
0000
38
00000001
00000000000000
68656c6c6f616c6c
00

So as you can see you can split up the packet into different type of values. You can normally just create a structure out of this and parse it to the structure but i was wondering if there are other ways.

The packet would look like this...

struct information 
{
 qint8 header
qint8 opcode
qint16 data
qint8 size
qint32 data
qint8 ID
qint64 data[7]
QString message
qint end
} ;

Someting like that i guess....

[SGaist]

Sounds like you should be using QDataStream and implement the related streaming operators for your structure.

[Jonnybravo]

is there a way to do it via array or do i have to build my own template that will cast a vector buffer and loop thru the vector base on the size and location i give it

[Taz742]

@Jonnybravo
not need array.
QDataStream are best way, for my point of view.
Here you can see how to use QDataStream
https://forum.qt.io/topic/84222/qdatastream-serialze-deserialize-myclass-pointer/2
It would be nice if you read everything in the end.

[VRonin]

Let's imagine your ce06000038000000010000000000000068656c6c6f616c6c00 is one 32bit signed integer and a QString.

QTcpSocket *socket = qobject_cast<QTcpSocket *>(socketObject);
if(!socket) return;
QDataStream streamer(socket);
streamer.startTransaction();
qint32 val1;
QString val2;
streamer >> val1 >> val2;
if(streamer.commitTransaction()){
qDebug() << "Read successful: " << val1 << val2;
}

[Jonnybravo]

The only issue with datastreams is that the data is always changing very rarely do you see the same data twice. So i understand that hes breaking up the data into variables.

qint32 val1;
QString val2;
streamer >> val1 >> val2;

Before in my C++ app i would create a unsigned char vector resize the vector to the recvd size of the socket create a template that would loop thru the position base on the value i sent to it.

its mostly about being able to recv the data verity it on my end and send back what i need.

[VRonin]

@Jonnybravo I don't think I understand what you mean

[SGaist]

Isn't the information structure what you are sending around ?

[Jonnybravo]

The information is not known i am just recving traffic from a tcp connection and trying to parse the data so that i can read it and check its validations on my end before replying back.

//Something like this i can do
//ce06000038000000010000000000000068656c6c6f616c6c00
qint8 header = recvdbuffer[0];
qint8 opcode = recvdbuffer[1];
qint8 size = recvdbuffer[4];
for(unsigned i = 0; i < recvdbuffer.length(); i++ )
{
// grab the string from the array
}

[SGaist]

How do you know when you received a full frame of data ?

[Jonnybravo]

The client builds the packet into one big buffer and sends it. On the server side i know the size and most data in the frame.

Qt just has a crap way of being able to have a dynamic way of extracting data from a socket.

The only methods i can think of off the top of my head is use qdatastream that is built on top of the data structure of what you are reading then pass the infomation into arguments later on to be used for whatever you need it for.

example:

QTcpSocket *socket = qobject_cast<QTcpSocket *>(socketObject);
if(!socket) return;
QDataStream streamer(socket);
streamer.startTransaction();

streamer >> val1 >> val2 >> val3 >> val4;

switch (val2)
case: // do what you need with val2 with a function that takes arguments val4 and val5
break;

[SGaist]

If your client is sending the information structure, why not implement the stream operators for that structure directly ? That way you'll have directly an information object to handle on the receiving end.

[VRonin]

You can always use startTransaction on the socket without using QDataStream

[Jonnybravo]

Could you tell me what im doing wrong here.. (qbyte array to qvector)

So ive created a class that gets data from a struct vector buffer

struct Packet
{
    public:
        QVector<quint8>Buffer;
        quint8 Header;
        quint8 OpCode;

public:
    Packet( ) {}
    Packet( int size ) : Buffer(size){ }

	/* Read values from packet */
    template <typename T> T Get( quint32 pos )
	{
		if( Buffer.size() < pos+sizeof(T) ) 
                      return 0;
		return *((T*)&Buffer[pos]);
	}
    quint8 GetByte( quint32 pos )
	{
        return Get<quint8>( pos );
	}
    quint16 GetWord( quint32 pos )
	{
        return Get<quint16>( pos );
	}
    unsigned GetDWord( quint32 pos )
	{
		return Get<unsigned>( pos );
	}
    quint64 GetQWord( quint32 pos )
	{
        return Get<quint64>( pos );
	}
    float GetFloat( quint32 pos )
	{
		return Get<float>( pos );
	}
    void GetString( quint32 pos, char* buffer )
	{	  
		unsigned strLen = GetDWord( pos );
		for(unsigned i = 0; i < strLen; i++ )
		{
            buffer[i] = Get<quint8>(pos+4+i);
		}
		buffer[strLen] = 0;
	}
};

So i am now trying to convert the data from qbyte to vector.

// Recv the entire buffer from bytesAvailable()
recvbuffer = socket->readAll();
Packet m_pak(recvbuffer.size());

QDataStream out(&recvbuffer ,QIODevice::WriteOnly);
out << m_pak.Buffer;

//doesnt fill the buffer with the qbytearray..

Now should i just instead of using datastream just resize it and insert the qbytearray into the vector.

recvbuffer = socket->readAll();
Packet m_pak(MAX_BUFFER);
m_pak.Buffer.resize(m_pak.Buffer.size() + recvbuffer.size());
memcpy(&m_pak.Buffer[m_pak.Buffer.size() - recvbuffer.size()], &recvbuffer[0], recvbuffer.size() * sizeof(int));

doesnt work..

recvbuffer = socket->readAll();
Packet m_pak(MAX_BUFFER);
m_pak.Buffer.reserve(m_pak.Buffer.size() + recvbuffer.size());
qCopy(&recvbuffer[0], &recvbuffer[recvbuffer.size()], std::back_inserter(m_pak.Buffer));
//crashes due to the inserter copy.

Any ideas in how to fix this or what i should do...

[SGaist]

How do you know you received the full data ?

[Jonnybravo]

i do a checksum on the packet size and of that of the header. if the data matches to what it should be i parse it..

any sugguestions how to fix my issue?