Qdatastreams and binary files.

jsulm

@Styx said in Qdatastreams and binary files.:

is there a way to loop thru the bytearray without using mid?

Sure (https://doc.qt.io/qt-5/qbytearray.html):
QByteRef operator[](int i)
char operator[](int i) const
char operator[](uint i) const
QByteRef operator[](uint i)

So

for (int i = 0; i < 3; ++i)
    bytearray[i];

Styx

So once i readall the file into the qbytearray i would use mid to break up the byte offset and copy them to another bytearray?

Is there away to get around not having to use so many qbytearray to copy data?

How would seek and read work from a Qfile?

02 - book count 

00 00 00 00 00 00 00 - (padding)

fb 2b 7d 13 - bookhash

09 - bookhashid

00 00 00 00 00 00 00 - (padding)

44 6e 49 4f 43 44 61 62 64 - booktype

42 - booktypeid

00 00 00 00 00 00 00 - (padding)

00 00 00 00 00 00 00 00 00 00 00 00 - bookDir and bookFileName (Qstring)

00 00 00 00 00 00 00 - (padding)

1a 10 a2 ae - bookhash

08 - bookhashid

00 00 00 00 00 00 00 

41 6S 64 4f 47 61 49 44 - booktype

3c - booktypeid

00 00 00 00 00 00 00 

00 00 00 00 00 00 00 00 00 00 00 00 - bookDir and bookFileName (Qstring)

This is a example of a file i am reading i was trying to read it into a struct but not sure if that is the correct method. Qdatastreams cant be used because the file wasn't written by qdatastreams.

as you seen in the code the book count is looped base on the same information provided.

mrjj

Hi
What app produces the file ?
Its not open source so you could get the actual record definitions ?

Christian Ehrlicher

@Styx said in Qdatastreams and binary files.:

Is there away to get around not having to use so many qbytearray to copy data?

Work with a plain const char * pointer

JKSH

@Christian-Ehrlicher said in Qdatastreams and binary files.:

Work with a plain const char * pointer

To add to @Christian-Ehrlicher's point: CallQByteArray:data() or QByteArray::constData() to get a raw pointer to your data. Then, you can use pointer arithmetic to extract your data.

QByteArray ba = file.readAll();
const char* data = ba.constData();

// Assuming that your file is little-endian...
memcpy(&m_binaryVersion, data +  0, sizeof(quint8 ));
memcpy(&bookCount,       data +  5, sizeof(quint8 ));
memcpy(&bookHash,        data + 13, sizeof(quint32));

EDIT: Code above changed from reinterpret_cast<> to memcpy() for cross-platform safety

Styx

@JKSH Since .data() is null terminated. Think it would be better to use shift left.

// Assuming that your file is little-endian...
m_binaryVersion = *reinterpret_cast<const quint8* >(data +  0 ) >> 8;
bookCount       = *reinterpret_cast<const quint8* >(data +  5) >> 12;
bookHash        = *reinterpret_cast<const quint32*>(data + 13) >> 16;

// example
bookCount=256 the first byte is '\0' then all the rest will be undetermined.

Shouldn't have issues calling the index and then looping thru the qbytearray to print out the data as well.

Styx

I have to read some where around 2000 binary files non of them the same but some contain same data.

How would i use seek and read dynamically to read each file. (Qfile api).

JKSH

@Styx said in Qdatastreams and binary files.:

@JKSH Since .data() is null terminated. Think it would be better to use shift left.

m_binaryVersion = *reinterpret_cast<const quint8* >(data +  0 ) >> 8;
bookCount       = *reinterpret_cast<const quint8* >(data +  5) >> 12;
bookHash        = *reinterpret_cast<const quint32*>(data + 13) >> 16;

// example
bookCount=256 the first byte is '\0' then all the rest will be undetermined.

I don't get it. Could you please explain how this works?

@Styx said in Qdatastreams and binary files.:

How would i use seek and read dynamically to read each file. (Qfile api).

Take the code that reads one file and put it in a loop. Pass a different filename each loop iteration.

JonB

@Styx said in Qdatastreams and binary files.:

@JKSH Since .data() is null terminated. Think it would be better to use shift left.

// Assuming that your file is little-endian...
m_binaryVersion = *reinterpret_cast<const quint8* >(data +  0 ) >> 8;
bookCount       = *reinterpret_cast<const quint8* >(data +  5) >> 12;
bookHash        = *reinterpret_cast<const quint32*>(data + 13) >> 16;

// example
bookCount=256 the first byte is '\0' then all the rest will be undetermined.

I don't know what you're trying to achieve here (as @JKSH said), but:

You are using shift right, not left.
*reinterpret_cast<const quint8* >(data + 0 ) returns a quint8. Since that is (unsigned) 8-bits in size, >> 8 always returns 0 regardless of content.
Similarly for *reinterpret_cast<const quint8* >(data + 5) >> 12, except that >> 12 makes even less sense for an 8-bit value.
QByteArray:data() is indeed (extra) \0 terminated, but that has no relevance to any of the lines of code you wrote.

The code without any shifts written by @JKSH makes sense. I'm afraid yours does not!

JonB

@JKSH said in Qdatastreams and binary files.:

bookHash = *reinterpret_cast<const quint32*>(data + 13);

Have you actually tried this line? Because I would assume it will "segment fault" (or whatever, probably something else). You are trying to dereference a 32-bit int from data + 13, which will be an odd numbered address. Whoops! :) [I have a feeling static_cast<> would warn/prohibit this at compile-time?]

You must be very careful recommending to treat a binary block like this as though you can index into it directly for the types you know were serialized there, for this kind of reason. Here you need to pull the 4 bytes out of the buffer (e.g. memcpy() directly into an &quint32 if you know endian-ness is same on host as in file), or some other safe approach.

JKSH

@JonB said in Qdatastreams and binary files.:

@JKSH said in Qdatastreams and binary files.:

bookHash = *reinterpret_cast<const quint32*>(data + 13);

Have you actually tried this line? Because I would assume it will "segment fault" (or whatever, probably something else). You are trying to dereference a 32-bit int from data + 13, which will be an odd numbered address. Whoops! :)

Thanks for the heads-up. I tried compiling it using MinGW 7.3.0 32-bit, MSVC 2017 32-bit, and MSVC2017 64-bit (all with Qt 5.14.0, release mode) and got the expected results every time. However, your comment prompted me to do some digging which led me to this question: Should I worry about the alignment during pointer casting?

I'll update my sample code.

[I have a feeling static_cast<> would warn/prohibit this at compile-time?]

Static casting cannot be used to convert a byte array into an integer at all, no matter where the bytes sit in memory.

Christian Ehrlicher

@JKSH said in Qdatastreams and binary files.:

You are trying to dereference a 32-bit int from data + 13, which will be an odd numbered address.

This is working fine on x86_64, only slow. It does not work on some ARM processors, see e.g. here: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka15414.html

JonB

@JKSH , @Christian-Ehrlicher
Very interesting! I thought processors just "bus-dumped" or whatever on an odd address, I didn't know they would "trap" the alignment and "recover", and thereby work but run slowly. I wonder what the last "friendly" processor architecture I saw --- Motorola 68000 family, like 68010 or 68020, not this x86-type stuff --- would have done? :)

Styx

So I have some 3000 files to go through and read. Currently I have been indexof and mid to find strings and variables.

QByteArray filedata = file.readall();
int j = 0;
while ((j = filedata.indexOf("books", j)) != -1) {
    QDegub ()  << "Found String  index position " << j ;
    ++j;
// put the qbytearray into a qstring
}

This method can get ugly as some of the files have over 50 strings inside it and this would make the source code look ugly.

Should i just seek to the start position then read from that point on? Should i use readline? Or read? qbytearray readall then store it in another buffer. Is there a way to extract strings from a qbytearray?

kuzulis

Stop, guys... As I remember, you can read a simple data types (int, uint and etc) using the QDataStream. And even own structures, which are not written by QDataStream (use raw read for this). You even can read a strings as a RAW objects.

Styx

@kuzulis You explain what your talking about?

Always thought Qdatastreams couldn't parse padding structures and that you could only read and write from it if it was done by qt.

Mind showing a example?

JonB

@Styx
Despite what @kuzulis has written, it does not follow that you can use QDataStream to deserialize, say, an int, even if there is no padding at all in its serialization (I don't know what QDataStream does or does not put in). The point is you have said that the file format you are trying to read is produced by someone else, not using QDataStream to serialize, right? In that case, as an example, even if it outputs, say, a 32-bit int in 4 bytes you do not know whether that means low->high or high->low bytes. And nor does QDataStream. So how can that correctly deserialize if the way it was saved differs from however QDataStream int-order deserialization works?

EDIT See @kuzulis's code below which shows that you must tell QDataStream which order to expect if the output was not produced with the default QDataStream order, which then allows you to proceed.

kuzulis

What? Here an example e.g. how to parse a WAV file using the QDataStream:

...
const quint32 kRiffId = 0x52494646;
const quint32 kWaveId = 0x57415645;
const quint32 kFmtId = 0x666d7420;
const quint32 kPcmFmtSize = 16; // for PCM only
const quint16 kAudioFormatId = 1; // WAVE_FORMAT_PCM
const quint32 kDataId = 0x64617461;

...
    bool readFormat()
    {
        file.seek(0);

        QDataStream in(&file);

        quint32 chunkId = 0;
        in.setByteOrder(QDataStream::BigEndian);
        in >> chunkId;
        if (chunkId != kRiffId) // "RIFF"
            return false;

        quint32 chunkSize = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> chunkSize; // file size

        quint32 formatId = 0;
        in.setByteOrder(QDataStream::BigEndian);
        in >> formatId;
        if (formatId != kWaveId) // "WAVE"
            return false;

        quint32 subchunk1Id = 0;
        in.setByteOrder(QDataStream::BigEndian);
        in >> subchunk1Id;
        if (subchunk1Id != kFmtId) // "fmt "
            return false;

        quint32 subchunk1Size = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> subchunk1Size;
        if (subchunk1Size != kPcmFmtSize) // for PCM format only
            return false;

        quint16 audioFormat = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> audioFormat;
        if (audioFormat != kAudioFormatId) // for PCM format only
            return false;

        quint16 numChannels = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> numChannels;
        if (numChannels == 0)
            return false;

        quint32 sampleRate = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> sampleRate;
        if (sampleRate == 0)
            return false;

        quint32 byteRate = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> byteRate;

        quint16 blockAlign = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> blockAlign;
        if (blockAlign == 0)
            return false;

        quint16 bitsPerSample = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> bitsPerSample;
        if (bitsPerSample == 0)
            return false;

        quint32 subchunk2Id = 0;
        in.setByteOrder(QDataStream::BigEndian);
        in >> subchunk2Id;
        if (subchunk2Id != kDataId) // "data"
            return false;

        quint32 subchunk2Size = 0;
        in.setByteOrder(QDataStream::LittleEndian);
        in >> subchunk2Size;
        if (subchunk2Size == 0)
            return false;

        startDataOffset = sizeof(chunkId)
                + sizeof(chunkSize)
                + sizeof(formatId)
                + sizeof(subchunk1Id)
                + sizeof(subchunk1Size)
                + sizeof(audioFormat)
                + sizeof(numChannels)
                + sizeof(sampleRate)
                + sizeof(byteRate)
                + sizeof(blockAlign)
                + sizeof(bitsPerSample)
                + sizeof(subchunk2Id)
                + sizeof(subchunk2Size);

        format.setCodec(QLatin1String(kAudioCodec));
        format.setChannelCount(numChannels);
        format.setSampleRate(sampleRate);
        format.setSampleSize(bitsPerSample);
        format.setSampleType(QAudioFormat::SignedInt); // TODO: This is correctly?
        format.setByteOrder(QAudioFormat::LittleEndian);

        return file.seek(startDataOffset);
    }

JonB

@kuzulis
Sorry, what I meant was, this works because you know at each stage whether you expect little-endian or big-endian order in the external data stream format, and you explicitly code to tell QDataStream that. I meant that a pure call to whichever way round the default of QDataStream expects will not work, whereas it would if QDataStream (with default settings) had also been used at the output side.

I won't delete my post now, but I will amend it to point out that what you have written allows it to be accomplished.

Styx

Far as the format goes its LittleEndian (binary)... goes string to value throught the entire file. Somethings are confusing what about a uknown format? What is startDataOffset?