Decoding QByteArray into UTF-16 and UInt32

jars121

I'm trying to read data from a UDP connection. I'm able to bind to the IP address and port successfully, and have the readyRead() callback function all operating as expected.

As part of the initiation, I send a packet to the UDP server, which responds with some handshake data. The handshake data is 408 bytes (per the UDP server documentation), which I've verified. The 408 bytes are specified as follows:

50 char UTF-16 (50*2 bytes)
50 char UTF-16 (50*2 bytes)
Int (4 bytes)
Int (4 bytes)
50 char UTF-16 (50*2 bytes)
50 char UTF-16 (50*2 bytes)

All bytes are Little Endian. The issue I'm having is in decoding the bytes into strings (for the UTF-16 bytes) and integers (for the ints). Below is my code (I've removed some of the UDPSocket setup as it's not relevant).

while (udpSocket->hasPendingDatagrams()) {
    QHostAddress sender = QHostAddress(IPAddr);
    quint16 port = udpPort;

    QByteArray buf(udpSocket->pendingDatagramSize(), Qt::Uninitialized);
    udpSocket->readDatagram(buf.data(), buf.size(), &sender, &udpPort);

    qDebug() << buf.mid(0, 100).toHex();
    qDebug() << buf.mid(200, 4).toHex();
}

How can I modify the second last line to print the first UTF-16 packet as a string and the last line to print the first int packet as an int? I've got toHex() there as an example.

jsulm

@jars121
https://doc.qt.io/qt-5/qstring.html#fromUtf16
https://doc.qt.io/qt-5/qbytearray.html#toInt

hskoglund

Hmm, first link looks good, but 2nd link: using toInt() will most likely fail, because it assumes that the QByteArray contents was generated with something like QByteArray::number().

I'm guessing the handshake integer data is raw (not formatted as a string), here's an example:

QByteArray s("1234"); // this works with toInt()
QByteArray n("\u0001\u0002\u0003\u0004");  // but I'm guessing this flavor

int i1 = s.toInt();
int i2 = n.toInt();

qDebug() << i1 << i2;

memcpy(&i2,n.data(),4);  // so, use this style of conversion
qDebug() << i2;

// which is the same as this (1,2,3,4 Little Endian four bytes)
qDebug() << 4 * 256*256*256 + 3 * 256*256 + 2 * 256 + 1;

jsulm

@hskoglund @jars121 Better/cleaner solution would be to use https://doc.qt.io/qt-5/qdatastream.html and overload needed QDataStream &QDataStream::operator>>(...) operators.

jsulm

@hskoglund said in Decoding QByteArray into UTF-16 and UInt32:

memcpy(&i2,n.data(),4); // so, use this style of conversion

Don't forget about endianness :-)

aha_1980

@jars121 as @jsulm already said, your problem sounds like an excellent case for QDataStream.

Regards

jars121

Thanks everyone, much appreciated as always.

I 've got the below amended QDataStream approach, but I've not used this method before so am not quite getting it.

while (udpSocket->hasPendingDatagrams()) {
    QHostAddress sender = QHostAddress(IPAddr);
    quint16 port = udpPort;

    QByteArray buf(udpSocket->pendingDatagramSize(), Qt::Uninitialized);
    QDataStream str(&buf, QIODevice::ReadOnly);
    udpSocket->readDatagram(buf.data(), buf.size(), &sender, &udpPort);

    str.setByteOrder(QDataStream::LittleEndian);

    //How do I get the first UTF-16 value as a string here?
    quint16 firstString;
    str >> firstString;
}

I get the logic to the above, but I don't see how I can break the QDataStream into its constituent packets (i.e. 4x 100 byte 50-character UTF-16s and 2x 32-bit ints) like I can do with the buf.mid() approach. Do I need to str.readBytes() or str.readRawData() beforehand?

jars121

Ok I've been playing with this, and seem to have worked out a solution using the original QByteArray approach which I've listed below. I'm really looking for the optimal approach however, so if the QDataStream approach is better I'm more than happy to use that, but I'll need some guidance as I've not yet had any success.

while (udpSocket->hasPendingDatagrams()) {
    QHostAddress sender = QHostAddress(IPAddr);
    quint16 port = udpPort;

    QByteArray buf(udpSocket->pendingDatagramSize(), Qt::Uninitialized);
    udpSocket->readDatagram(buf.data(), buf.size(), &sender, &udpPort);

    QByteArray first;
    QByteArray second;

    first.append(buf.mid(0, 100));
    second.append(buf.mid(100, 100));

    QString strFirst = QString::fromUtf16(reinterpret_cast<const ushort*>(first.constData());
    QString strSecond = QString::fromUtf16(reinterpret_cast<const ushort*>(second.constData());
}

The above seems to work nicely, but I can't help but feel that using a QByteArray for each parameter in the packet is exactly what the QDataStream approach would be doing for me :D

Edit: Also, I can't seem to get the same approach to work for the ints. The QByteArray::toInt method always results in 0.

jars121

Ok, I'm able to decode all 6 parameters, but I can't imagine the below approach is the optimal way to do so.

while (udpSocket->hasPendingDatagrams()) {
    QHostAddress sender = QHostAddress(IPAddr);
    quint16 port = udpPort;

    QByteArray buf(udpSocket->pendingDatagramSize(), Qt::Uninitialized);
    udpSocket->readDatagram(buf.data(), buf.size(), &sender, &udpPort);

    QByteArray first;
    QByteArray second;
    QByteArray third;
    QByteArray fourth;
    QByteArray fifth;
    QByteArray sixth;

    first.append(buf.mid(0, 100));
    second.append(buf.mid(100, 100));
    third.append(buf.mid(200, 4));
    fourth.append(buf.mid(204, 4));
    fifth.append(buf.mid(208, 100));
    sixth.append(buf.mid(308, 100));

    QString stringFirst = QString::fromtUtf16(reinterpret_cast<const ushort*>(first.constData()));
    QString stringSecond = QString::fromtUtf16(reinterpret_cast<const ushort*>(second.constData()));

    int intThird = third.toInt();
    memcpy(&intThird, third.data(), 4);

    int intFourth = fourth.toInt();
    memcpy(&intFourth, fourth.data(), 4);

    QString stringFifth = QString::fromtUtf16(reinterpret_cast<const ushort*>(fifth.constData()));
    QString stringSixth = QString::fromtUtf16(reinterpret_cast<const ushort*>(sixth.constData()));
}

I'll have another play with a QDataStream approach now, as I'm sure the above can be simplified with a single QDataStream object.

aha_1980

@jars121,

I think your problem are the 50 char UTF-16. I guess these are no QStrings, therefore you cannot directly serialize/deserialize them. For the int that's no problem, QDataStream can handle that.

I wonder, if you can set up a type using UnicodeArray = std::array<QChar, 50>; and use that to read from the data stream?

Untested example:

QDataStream in(&udpSocket);  
UnicodeArray arr0;
UnicodeArray arr1;
qint32 a;
qint32 b;
UnicodeArray arr2;
UnicodeArray arr3;
in >> arr0 >> arr1 >> a >> b >> arr2 >> arr3;

Now you would just need to convert the UnicodeArray into QString.

Regards

jars121

@aha_1980 I'm afraid I'm getting the following error:

invalid operands to binary expression ('QDataStream' and 'std::array<QChar, 50>')

Your earlier assumption is correct, the 50 char UTF-16 are not QStrings. The reinterpret_cast approach I posted earlier does work for decoding, but I'd rather use QDataStream instead of having a separate QByteArray for each parameter.

I feel like I've tried every permutation with the QDataStream approach but I'm not getting the expected output. Even sizeof() testing of the various components doesn't return the correct result for some reason.

jars121

Ok so I've made some progress; the below correctly prints the first packet from the QDataStream:

ushort packet1[50];
ushort packet2[50];
quint32 packet3;
quint32 packet4;
ushort packet5[50];
ushort packet6[50];

uint16_t i = 0;
while (str.atEnd() == false) {
    str >> packet1[i];
    i++;
}

qDebug() << QString::fromUtf16(reinterpret_cast<const ushort*>(packet1));

How can I expand the while loop operation(s) to put the QDataStream (str) data into the remain ushorts and quint32s? I really don't understand why a simple str >> packet1 >> packet2 >> etc. operation doesn't work?

hskoglund

@jars121 said

I really don't understand why a simple str >> packet1 >> packet2 >> etc. operation doesn't work?

Because while QDataStream is powerful, it does not know about packets of 50 ushorts. A single ushort, no problem. But old C-style arrays, e.g. *ushort packet1[50] are unknown territory :-(

Instead, QDataStream supports arrays in the flavors of QList, QVector etc. If you use QVector, you can do str >> myVector, voila.

But (there's always a but) for you to be able use QVector instead of ushort packet1[50], it needs to arrive in the correct format over the wire/UDP: the 50 ushorts has to be prefixed with a length/count integer. If you cannot persuade whatever entity is handing you the ushorts over the UDP connection to prefix them with a length integer, you're a bit out of luck. But I would suggest you use QVectors anyway, say like this:

#include "qvector.h"

// declare the packets
QVector<ushort> packet1(50);
QVector<ushort> packet2(50);
quint32 packet3;
quint32 packet4;
QVector<ushort> packet5(50);
QVector<ushort> packet6(50);

str.setByteOrder(QDataStream::LittleEndian);

for (auto &u : packet1)
    str >> u;
QString s1 = QString::fromUtf16(packet1.data());

for (auto &u : packet2)
    str >> u;
QString s2 = QString::fromUtf16(packet2.data());

str >> packet3;
str >> packet4;

for (auto &u : packet5)
    str >> u;
QString s5 = QString::fromUtf16(packet5.data());

for (auto &u : packet6)
    str >> u;
QString s6 = QString::fromUtf16(packet6.data());

(I.e. instead of str >> packet1; you can use a for loop, only one extra line).