Qdatastreams and binary files.
-
I'm trying read binary files using qdatastreams then parsing them to structures. Soon as i parse the first byte i end up getting the wrong information after it.
#ifndef DATA_H #define DATA_H #include "mainwindow.h" struct BinaryType { quint8 m_binaryVersion; friend QDataStream& operator>>(QDataStream &in, BinaryType &p); }; QDataStream &operator>>(QDataStream &in, BinaryType &p) { in >> p.m_binaryVersion; return in; } struct Books { quint8 bookCount; quint32 bookHash; quint8 bookHashId; QString bookType; quint8 bookTypeId; QString bookDir; QString bookFileName; friend QDataStream& operator>>(QDataStream &in,Books &p); }; QDataStream &operator>>(QDataStream &in, Books &p) { in >> p.bookCount; in >> p.bookHash; in >> p.bookHashId; in >> p.bookType; in >> p.bookTypeId; in >> p.bookDir; in >> p.bookFileName; return in; } #endif // DATA_H
So i open my file then call the struct then call the datastream to parse the file...
QFile file(fileName); if ( file.open( QIODevice::ReadOnly) ) { qDebug() << "Opened:" << file.fileName() << endl; BinaryType binInfo; Books booksInfo; QDataStream in(&file); in >> binInfo.m_binaryVersion; in >> booksInfo.bookCount >> booksInfo.bookHash >> booksInfo.bookHashId >> booksInfo.bookType >> booksInfo.bookTypeId >> booksInfo.bookDir >> booksInfo.bookFileName; qDebug() << "BinaryType Struct" << binInfo.m_binaryVersion; qDebug() << "Books Struct" << booksInfo.bookCount << booksInfo.bookHash << booksInfo.bookHashId << booksInfo.bookType << booksInfo.bookTypeId << booksInfo.bookDir << booksInfo.bookFileName; } else { qDebug() << "Failed:" << file.fileName() << file.errorString() << endl; return; } }
So the binary file looks like this.
0x01 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xcb 0x3b 0x8d
0x12 0x09 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x41 0x7e 0x68 0x5e 0x52 0x73 0x60
0x71 0x73 0x41 0x00 0x00 0x00 0x00 0x00 0x00 (hex bookDir and hex bookFileName)my program prints out 0x01 then 0x00 etc... never gets the correct data....
its suppose to be...
m_binaryVersion = 0x01
bookCount = 0x02
bookhash = 0xcb 0x3b 0x8d 0x12
bookhashId = 0x09
booktype = 0x41 0x7e 0x68 0x5e 0x52 0x73 0x60 0x71 0x73
booktypeid = 0x41
bookDir = hex etc...
bookFileName = hex etc..Also is there a way i can loop thru a qdatastream twice? since the book count is 2 and contains the same structure again?
Or should i just use readRawData to read it into chunks and parse it from there?
-
When your file was not saved with QDataStream you won't be able to read it with QDataStream.
-
I'm trying read binary files using qdatastreams then parsing them to structures. Soon as i parse the first byte i end up getting the wrong information after it.
#ifndef DATA_H #define DATA_H #include "mainwindow.h" struct BinaryType { quint8 m_binaryVersion; friend QDataStream& operator>>(QDataStream &in, BinaryType &p); }; QDataStream &operator>>(QDataStream &in, BinaryType &p) { in >> p.m_binaryVersion; return in; } struct Books { quint8 bookCount; quint32 bookHash; quint8 bookHashId; QString bookType; quint8 bookTypeId; QString bookDir; QString bookFileName; friend QDataStream& operator>>(QDataStream &in,Books &p); }; QDataStream &operator>>(QDataStream &in, Books &p) { in >> p.bookCount; in >> p.bookHash; in >> p.bookHashId; in >> p.bookType; in >> p.bookTypeId; in >> p.bookDir; in >> p.bookFileName; return in; } #endif // DATA_H
So i open my file then call the struct then call the datastream to parse the file...
QFile file(fileName); if ( file.open( QIODevice::ReadOnly) ) { qDebug() << "Opened:" << file.fileName() << endl; BinaryType binInfo; Books booksInfo; QDataStream in(&file); in >> binInfo.m_binaryVersion; in >> booksInfo.bookCount >> booksInfo.bookHash >> booksInfo.bookHashId >> booksInfo.bookType >> booksInfo.bookTypeId >> booksInfo.bookDir >> booksInfo.bookFileName; qDebug() << "BinaryType Struct" << binInfo.m_binaryVersion; qDebug() << "Books Struct" << booksInfo.bookCount << booksInfo.bookHash << booksInfo.bookHashId << booksInfo.bookType << booksInfo.bookTypeId << booksInfo.bookDir << booksInfo.bookFileName; } else { qDebug() << "Failed:" << file.fileName() << file.errorString() << endl; return; } }
So the binary file looks like this.
0x01 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xcb 0x3b 0x8d
0x12 0x09 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x41 0x7e 0x68 0x5e 0x52 0x73 0x60
0x71 0x73 0x41 0x00 0x00 0x00 0x00 0x00 0x00 (hex bookDir and hex bookFileName)my program prints out 0x01 then 0x00 etc... never gets the correct data....
its suppose to be...
m_binaryVersion = 0x01
bookCount = 0x02
bookhash = 0xcb 0x3b 0x8d 0x12
bookhashId = 0x09
booktype = 0x41 0x7e 0x68 0x5e 0x52 0x73 0x60 0x71 0x73
booktypeid = 0x41
bookDir = hex etc...
bookFileName = hex etc..Also is there a way i can loop thru a qdatastream twice? since the book count is 2 and contains the same structure again?
Or should i just use readRawData to read it into chunks and parse it from there?
-
How should i go about reading the data then? Qfile read or qbytearray readall then cover the data from there? Readrawdata?
A small example would be helpfull?
@Styx There is an example in the documentation: https://doc.qt.io/qt-5/qdatastream.html
QFile file("file.dat"); file.open(QIODevice::ReadOnly); QDataStream in(&file); // read the data serialized from the file QString str; qint32 a; in >> str >> a;
-
@Styx said in Qdatastreams and binary files.:
Qfile read or qbytearray readall then cover the data from there?
Yes
@jsulm: the data is not created with QDataStream, so reading with QDataStream will not (really) work.
-
@Styx said in Qdatastreams and binary files.:
Qfile read or qbytearray readall then cover the data from there?
Yes
@jsulm: the data is not created with QDataStream, so reading with QDataStream will not (really) work.
@Christian-Ehrlicher Ah, OK. I was thinking other way around - it is early morning here :-)
-
How should i go about reading the data then? Qfile read or qbytearray readall then cover the data from there? Readrawdata?
A small example would be helpfull?
@Styx said in Qdatastreams and binary files.:
How should i go about reading the data then?
First, you need to clarify:
- What is the binary layout of your file?
- Are the numbers in your file stored in little-endian or big-endian format?
- How are your text strings encoded? ASCII? UTF-8? Something else?
@Styx said in Qdatastreams and binary files.:
So the binary file looks like this.
0x01 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xcb 0x3b 0x8d
0x12 ....its suppose to be
m_binaryVersion = 0x01
bookCount = 0x02
bookhash = 0xcb 0x3b 0x8d 0x12OK, so your file layout is something like this...
Byte Offset Content 0 m_binaryVersion
1-4 ??? 5 bookCount
6-12 ???? 13-16 bookHash
There are 4 null bytes between
0x01
and0x02
, and there are 7 null bytes between0x02
and0xcb 0x3b 0x8d 0x12
. What are these null bytes for?Is
bookCount
supposed to occupy Bytes 5-12 as a qint64? -
how do i read chunks or sections with qbytearray? Should i store it in one buffer then read the section with qfile read?
1-4 bytes are just padding i believe 5 is the ammount of books that is written to the struct
0xcb 0x3b 0x8d is the bookhash
0x12 is the bookhashId
-
how do i read chunks or sections with qbytearray? Should i store it in one buffer then read the section with qfile read?
1-4 bytes are just padding i believe 5 is the ammount of books that is written to the struct
0xcb 0x3b 0x8d is the bookhash
0x12 is the bookhashId
@Styx said in Qdatastreams and binary files.:
how do i read chunks or sections with qbytearray?
-
how do i read chunks or sections with qbytearray? Should i store it in one buffer then read the section with qfile read?
1-4 bytes are just padding i believe 5 is the ammount of books that is written to the struct
0xcb 0x3b 0x8d is the bookhash
0x12 is the bookhashId
@Styx said in Qdatastreams and binary files.:
1-4 bytes are just padding
OK
i believe 5 is the ammount of books that is written to the struct
So the maximum number of books is 255?
What are bytes 6-12?
-
The null bytes are just null padding. max number of books are 5 each book has its own bookhash and bookhashId booktype bookdir and bookfilename
Would use qbytearray mid grab the max number of books which is 5 then loop thru the rest of the qbytearray to grab each bookinformation.
-
The null bytes are just null padding. max number of books are 5 each book has its own bookhash and bookhashId booktype bookdir and bookfilename
Would use qbytearray mid grab the max number of books which is 5 then loop thru the rest of the qbytearray to grab each bookinformation.
@Styx said in Qdatastreams and binary files.:
The null bytes are just null padding. max number of books are 5 each book has its own bookhash and bookhashId booktype bookdir and bookfilename
OK
Would use qbytearray mid grab the max number of books which is 5 then loop thru the rest of the qbytearray to grab each bookinformation.
Sounds good
-
is there a way to loop thru the bytearray without using mid?
was using...
Books bookinfo; bookinfo.bookCount = (bytearray.at(5) & 0xFFFF);
@Styx said in Qdatastreams and binary files.:
is there a way to loop thru the bytearray without using mid?
Sure (https://doc.qt.io/qt-5/qbytearray.html):
QByteRef operator[](int i)
char operator[](int i) const
char operator[](uint i) const
QByteRef operator[](uint i)So
for (int i = 0; i < 3; ++i) bytearray[i];
-
So once i readall the file into the qbytearray i would use mid to break up the byte offset and copy them to another bytearray?
Is there away to get around not having to use so many qbytearray to copy data?
How would seek and read work from a Qfile?
02 - book count 00 00 00 00 00 00 00 - (padding) fb 2b 7d 13 - bookhash 09 - bookhashid 00 00 00 00 00 00 00 - (padding) 44 6e 49 4f 43 44 61 62 64 - booktype 42 - booktypeid 00 00 00 00 00 00 00 - (padding) 00 00 00 00 00 00 00 00 00 00 00 00 - bookDir and bookFileName (Qstring) 00 00 00 00 00 00 00 - (padding) 1a 10 a2 ae - bookhash 08 - bookhashid 00 00 00 00 00 00 00 41 6S 64 4f 47 61 49 44 - booktype 3c - booktypeid 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - bookDir and bookFileName (Qstring)
This is a example of a file i am reading i was trying to read it into a struct but not sure if that is the correct method. Qdatastreams cant be used because the file wasn't written by qdatastreams.
as you seen in the code the book count is looped base on the same information provided.
-
Hi
What app produces the file ?
Its not open source so you could get the actual record definitions ? -
@Styx said in Qdatastreams and binary files.:
Is there away to get around not having to use so many qbytearray to copy data?
Work with a plain const char * pointer
-
@Styx said in Qdatastreams and binary files.:
Is there away to get around not having to use so many qbytearray to copy data?
Work with a plain const char * pointer
@Christian-Ehrlicher said in Qdatastreams and binary files.:
Work with a plain const char * pointer
To add to @Christian-Ehrlicher's point: Call
QByteArray:data()
orQByteArray::constData()
to get a raw pointer to your data. Then, you can use pointer arithmetic to extract your data.QByteArray ba = file.readAll(); const char* data = ba.constData(); // Assuming that your file is little-endian... memcpy(&m_binaryVersion, data + 0, sizeof(quint8 )); memcpy(&bookCount, data + 5, sizeof(quint8 )); memcpy(&bookHash, data + 13, sizeof(quint32));
EDIT: Code above changed from reinterpret_cast<> to memcpy() for cross-platform safety
-
@JKSH Since .data() is null terminated. Think it would be better to use shift left.
// Assuming that your file is little-endian... m_binaryVersion = *reinterpret_cast<const quint8* >(data + 0 ) >> 8; bookCount = *reinterpret_cast<const quint8* >(data + 5) >> 12; bookHash = *reinterpret_cast<const quint32*>(data + 13) >> 16; // example bookCount=256 the first byte is '\0' then all the rest will be undetermined.
Shouldn't have issues calling the index and then looping thru the qbytearray to print out the data as well.