QByteArray and QString does not match in byte size ?

[Dariusz]

Hey

Trying to wrap my head around bytes but I'm a tad lost...

Can any1 help me out understand why they don't match ?

QString dataA("21451254 125 32534 tgre34 r3re4v34gt24w efv42ew v")
QByteArray result;
QDataStream stream(&result, QIODevice::WriteOnly);
stream << dataA.toUtf8();

qDebug() << result.size() << dataA.size() << dataA.toUtf8().size();
//// 53 49 49 

The result.size() does not equal either dataA.size or dataA.toUtf8().size().
So I wonder how can I properly measure the data I want to store/send/retrieve?

Say I want to store the size of data I want to send in 1st location of bytearray so that when I receive it I can properly read data and extend as long as data is being received(tcp socket). But so far the qbytearray size does not equal the size of content? Or do we add 4 bytes by default for bytearray ?

Regards
Dariusz

[SGaist]

Hi,

The QDataStream operator of classes may add additional information needed to rebuild the object stream when reading it back. For example, size of vector, length of data stored in a QString or QByteArray etc.

[Kent-Dorfman]

Don't access results directly. Think of it as a memory backing for the stream. Since the purpose of QDataStream is mostly in platform independent serialization, some type metadata is also stored in the backing. Four bytes of extra length smells strongly like length of message and type metadata.

Try accessing the stored byte array using the readBytes() method on the stream and see what you get out of it.

Let us know!

[Dariusz]

Hmmm I did not get far with it... still struggling to solve the basics... here is the current code I try to run :

client.cpp

    /// Small data object
    QString dataA("21451254 125 32534 tgre34 r3re4v34gt24w efv42ew v");
    /// large data object
    QString dataB;
    for (int x = 0; x < 10000; ++x) {
        dataB += "23tgvrefbv4iknw23fgopq234;enwfo;nw2gno34wg3eorw;gn3eorw;'gn3ew";
    }

    QByteArray result;
    QDataStream stream(&result, QIODevice::WriteOnly);
    stream << 10 /// Packet idtype
           << dataA.size()
           << dataA.toUtf8()
           << dataB.size()
           << dataB.toUtf8();
    result.insert(0, result.size()); /// this is missing the extra overhead size?

    char *d;
    char *b;
    int totalSize, type, sizeA, bSize, sizeB;


    QDataStream streamX(&result, QIODevice::ReadOnly);
    streamX >> totalSize >> type; // read data for test
    mSocket->write(result);

    qDebug() << totalSize << result.size();
            ///620756992 620070 - why totalSize so much bigger than result.size() ?

The client sends the debug data above, no idea why the totalSize does not equal or its slightly over result.size() :- ( I though that result.size() would be about 4-x bytes more than totalSize but its wrong.

And here is server reading it :
server.cpp

if (!mReadingData) {
    mReadingData = true;
    QByteArray rec = socket->readAll(); /// read as much as we can - specify a limit in future maybe?
    mData += rec; /// add new data
    mTotalSize = 0; /// 0 out total size counter
    QDataStream streamX(&rec, QIODevice::ReadOnly);
    streamX >> mTotalSize;//// get the size of full file?
    mDataLoaded += rec.size(); /// increment what we added
    qInfo() << "Last MSG Size : " << rec.size(); /// how much bytes we got from last message?
    qInfo() << "Readed Total data size : " << mTotalSize; /// how much bytes we need in total ?

} else {
    qInfo()<<"Adding data";
    QByteArray rec = socket->readAll(); /// read next chunk of data?
    mData += rec;
    mDataLoaded += rec.size(); /// increment what we added
    qInfo() << "Last MSG Size : " << rec.size(); /// how much bytes we got from last message?
    qInfo() << "Readed Total data size : " << mTotalSize; /// how much bytes we need in total ?
    if (mDataLoaded == mTotalSize) { /// we loaded all data we send, now process the packet and unload the data in to proper object?
        processData(mData);
        mReadingData = false;
        mData.clear(); /// clear old data
    }
}
//qInfo() << "Bytes loaded : " << mDataLoaded;
qInfo() << " TCPConnection::bytesAvailable" << socket->bytesAvailable(); /// always returns 0 ?
qInfo() << " TCPConnection::bytesToWrite" << socket->bytesToWrite(); /// always returns 0 ?

on Server site when I receive 1st transfer I get
Last MSG Size : 14600 < thi seems ok, 1st chunk of data.
Readed Total data size : 2666130979571105792 < this is waaaaaaaaaaaaay off, it should have been 620756992 if I am to believe the totalSize from client side

And I'm lost. I must be checking or measuring something incorrectly here :- (

I also tried to read data size via streamer but that returns 0

    char* raw;
    uint length;
    streamX.readBytes(raw, length);
    qDebug() << "length = " << length;

[Christian Ehrlicher]

You're writing raw bytes into a stream which is encoded with QDataStream and expect you don't break the datastream? Strange...

[Dariusz]

@Christian-Ehrlicher You mean that if I do

  result.insert(0, result.size());

I break something and that I should use QDataStream to add in the size of the data ?

I'm very new to QDataStreams/QIODevice/QByteArrays tbh. Still getting a hang of it.

[Christian Ehrlicher]

QDataStream writes the data with it's own format into a stream/QByteArray. You just write something directly into the stream generated by QDateStream - so yes.

[SGaist]

@Dariusz said in QByteArray and QString does not match in byte size ?:

QByteArray result;
QDataStream stream(&result, QIODevice::WriteOnly);
stream << 10 /// Packet idtype
<< dataA.size()
<< dataA.toUtf8()
<< dataB.size()
<< dataB.toUtf8();
result.insert(0, result.size()); /// this is missing the extra overhead size?

dataA and dataB are QStrings which already has QDataStream operators so why are you trying to serialise them manually ?

[Dariusz]

@SGaist I think because I don't know that :- ) even tho I read about it. I think I'm being overwhelmed by qt features and working against the flow of water. For examples I decided to go back to basic "structures" say :

    unsigned char *ar;
    unsigned int n = 123123;
    ar[0] = (n >> 24) & 0xFF;
    ar[1] = (n >> 16) & 0xFF;
    ar[2] = (n >> 8) & 0xFF;
    ar[3] = n & 0xFF;

    qDebug() << ar[0] << ar[1] << ar[2] << ar[3];


    int num = 0;
    for (int i = 0; i < 4; i++) {
        num <<= 8;
        num |= ar[i];
    }

    qDebug() << num;

This serialize/deserialize properly.

But when I do

    QByteArray ar;
    ar.reserve(4);
    unsigned int n = 123123;
   .....

Instead, then it does not work and I'm at loss. Perhaps I really miss understanding how I am meant to use QByteArray somehow. :- (

Edit. doing more read on it, yeah it seems like QT does a lot more to help with my task than I expected and I'm pushing against the design...

[Kent-Dorfman]

This post is deleted!

[Kent-Dorfman]

Here's a complete example for you, including passing an array.

#include <iostream>
#include <iomanip>

#include <QByteArray>
#include <QDataStream>
#include <QString>

// =============================================================================
int main(int argc, char** argv) {

    using namespace std;

    QByteArray buffer;                      // serialization backing buffer
    QString aStr("No Way.....");            // a test string
    QString bStr("Way, dude");              // second test string
    int j(32768);                           // throw an integer in, too
    const char* str("an array");
    int l(strlen(str));

    cout << "--- input ---" << endl <<      // show stream input
            aStr.toStdString() << endl <<
            bStr.toStdString() << endl <<
            str << endl <<
            j << endl;

    // assume this end is the transmitter
    QDataStream serialize(&buffer, QIODevice::WriteOnly);
    serialize << aStr << bStr;
    serialize.writeBytes(str, l);
    serialize << j;         // serialize the data


    // SEND "buffer" as a network packet somewhere else, then read it from the socket

    // assume this end is on the receiver
    QDataStream deserialize(&buffer, QIODevice::ReadOnly);
    QString outStrA, outStrB;
    int i;
    unsigned outL;
    char* ptr;

    // pull the data out of the stream in FIFO order
    deserialize >> outStrA >> outStrB;
    deserialize.readBytes(ptr, outL);
    deserialize >> i;

     // show the output
    cout << endl << endl << "--- output ---" << endl <<
            outStrA.toStdString() << endl <<
            outStrB.toStdString() << endl <<
            ptr << "    --- length=" << outL << endl <<
            i << endl;

    delete [] ptr;

    return 0;

}

[Dariusz]

Hey @Kent-Dorfman, thanks for the example, it looks interesting. But I'm still lost :- ) Sorry...

In any case, after testing the char *ar example above, I went back to re-read more on QDataStream/QTcpSocket/QByteArrays, Seems that in latest Qt that entire system is more "connected", ie if I send large file, I no longer need to handle the size of the file/packet/data I want to send as 1st argument when sending it across QTcp as that is now handled by Qt. - unless I read incorrectly.

I've figured out this part lately, but now I moved to reading the data from QTcpSocket and that's where I seem to be stuck..

    QByteArray ar;
    QDataStream sendStream(&ar, QIODevice::ReadWrite);
    sendStream << quint16(0) << dataA << dataB;
    sendStream.device()->seek(0);  
    sendStream << (quint16)(ar.size() - sizeof(quint16)); // replace the previously allocated quint16(0) value

And now I'm stuck on socketReading it so

slot:

void networkTest::readyRead() {
     QTcpSocket *socket = static_cast<QTcpSocket *>(sender());
     if (!socket) return;
     QDataStream in(socket);
     in.startTransaction();
     in >> mData; /// mData = QByteArray - we no longer need size of data because QT handles it (???) so just add data to mData .h memher
     if (!in.commitTransaction()) { // check if end data, if no, return and wait for more data in next event loop 
         active();
         qDebug() << in->status(); // returns 1 every time ? :- (
         return;
    }
    /// never gets called 
    processReceivedData();

}

TIA

Ok apparently, this does not work, it breaks entire data in QByteArray...

    serialize.device()->seek(0);
    serialize << buffer.size() - sizeof(quint16);

...

This is fun rollercoaster. 1 step forward, 2 steps backwards.

Ok perhaps the seek(0) does work, I just need to stick to proper types..

    sendStream.device()->seek(0);
    sendStream << (int) (block.size() - sizeof(int));

cant use quint, and all that. Hmmm... 1 step forward... lets see what are the next 2 steps back :- )

EDIT 123123.
Ok I got it working, yeah esentially I need to ensure that whatever int I pass in int/quint/uint/uint8_t etc etc that they all match and are the same ints... say QByeteArray.size() return int, then I have to cast every other int in to (int) number to properly work. Even if updating existing value in array via stream/qiodevice seek...

Ahh this was fun. Thanks every one for help!