Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Qdatastreams and binary files.
Forum Updated to NodeBB v4.3 + New Features

Qdatastreams and binary files.

Scheduled Pinned Locked Moved Unsolved General and Desktop
36 Posts 7 Posters 5.0k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • StyxS Offline
    StyxS Offline
    Styx
    wrote on last edited by
    #1

    I'm trying read binary files using qdatastreams then parsing them to structures. Soon as i parse the first byte i end up getting the wrong information after it.

    #ifndef DATA_H
    #define DATA_H
    
    #include "mainwindow.h"
    
    struct BinaryType
    {
      quint8 m_binaryVersion;
      friend QDataStream& operator>>(QDataStream &in, BinaryType &p);
    };
    
    QDataStream &operator>>(QDataStream &in, BinaryType &p)
    {
        in >> p.m_binaryVersion;
        return in;
    }
    
    struct Books
    {
        quint8     bookCount;
        quint32    bookHash;
        quint8     bookHashId;
        QString    bookType;
        quint8     bookTypeId;
        QString    bookDir;
        QString    bookFileName;
        friend QDataStream& operator>>(QDataStream &in,Books &p);
    };
    
    QDataStream &operator>>(QDataStream &in, Books &p)
    {
        in >> p.bookCount;
        in >> p.bookHash;
        in >> p.bookHashId;
        in >> p.bookType;
        in >> p.bookTypeId;
        in >> p.bookDir;
        in >> p.bookFileName;
        return in;
    }
    
    #endif // DATA_H
    

    So i open my file then call the struct then call the datastream to parse the file...

    QFile file(fileName);
            if ( file.open( QIODevice::ReadOnly) )
            {
                qDebug() << "Opened:" << file.fileName() << endl;
                
                BinaryType binInfo;
                Books      booksInfo;
                
                QDataStream in(&file);
                
                in >> binInfo.m_binaryVersion;
                in >> booksInfo.bookCount >> booksInfo.bookHash >> booksInfo.bookHashId >> booksInfo.bookType >> booksInfo.bookTypeId >> booksInfo.bookDir >> booksInfo.bookFileName;
                
                qDebug() << "BinaryType Struct" << binInfo.m_binaryVersion;
                qDebug() << "Books Struct" << booksInfo.bookCount << booksInfo.bookHash << booksInfo.bookHashId << booksInfo.bookType << booksInfo.bookTypeId << booksInfo.bookDir << booksInfo.bookFileName;
                
            }
            else
            {
                qDebug() << "Failed:" << file.fileName() << file.errorString() << endl;
                return;
            }
        }
    

    So the binary file looks like this.

    0x01 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xcb 0x3b 0x8d
    0x12 0x09 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x41 0x7e 0x68 0x5e 0x52 0x73 0x60
    0x71 0x73 0x41 0x00 0x00 0x00 0x00 0x00 0x00 (hex bookDir and hex bookFileName)

    my program prints out 0x01 then 0x00 etc... never gets the correct data....

    its suppose to be...

    m_binaryVersion = 0x01
    bookCount = 0x02
    bookhash = 0xcb 0x3b 0x8d 0x12
    bookhashId = 0x09
    booktype = 0x41 0x7e 0x68 0x5e 0x52 0x73 0x60 0x71 0x73
    booktypeid = 0x41
    bookDir = hex etc...
    bookFileName = hex etc..

    Also is there a way i can loop thru a qdatastream twice? since the book count is 2 and contains the same structure again?

    Or should i just use readRawData to read it into chunks and parse it from there?

    JonBJ 1 Reply Last reply
    0
    • Christian EhrlicherC Offline
      Christian EhrlicherC Offline
      Christian Ehrlicher
      Lifetime Qt Champion
      wrote on last edited by
      #2

      When your file was not saved with QDataStream you won't be able to read it with QDataStream.

      Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
      Visit the Qt Academy at https://academy.qt.io/catalog

      1 Reply Last reply
      3
      • StyxS Styx

        I'm trying read binary files using qdatastreams then parsing them to structures. Soon as i parse the first byte i end up getting the wrong information after it.

        #ifndef DATA_H
        #define DATA_H
        
        #include "mainwindow.h"
        
        struct BinaryType
        {
          quint8 m_binaryVersion;
          friend QDataStream& operator>>(QDataStream &in, BinaryType &p);
        };
        
        QDataStream &operator>>(QDataStream &in, BinaryType &p)
        {
            in >> p.m_binaryVersion;
            return in;
        }
        
        struct Books
        {
            quint8     bookCount;
            quint32    bookHash;
            quint8     bookHashId;
            QString    bookType;
            quint8     bookTypeId;
            QString    bookDir;
            QString    bookFileName;
            friend QDataStream& operator>>(QDataStream &in,Books &p);
        };
        
        QDataStream &operator>>(QDataStream &in, Books &p)
        {
            in >> p.bookCount;
            in >> p.bookHash;
            in >> p.bookHashId;
            in >> p.bookType;
            in >> p.bookTypeId;
            in >> p.bookDir;
            in >> p.bookFileName;
            return in;
        }
        
        #endif // DATA_H
        

        So i open my file then call the struct then call the datastream to parse the file...

        QFile file(fileName);
                if ( file.open( QIODevice::ReadOnly) )
                {
                    qDebug() << "Opened:" << file.fileName() << endl;
                    
                    BinaryType binInfo;
                    Books      booksInfo;
                    
                    QDataStream in(&file);
                    
                    in >> binInfo.m_binaryVersion;
                    in >> booksInfo.bookCount >> booksInfo.bookHash >> booksInfo.bookHashId >> booksInfo.bookType >> booksInfo.bookTypeId >> booksInfo.bookDir >> booksInfo.bookFileName;
                    
                    qDebug() << "BinaryType Struct" << binInfo.m_binaryVersion;
                    qDebug() << "Books Struct" << booksInfo.bookCount << booksInfo.bookHash << booksInfo.bookHashId << booksInfo.bookType << booksInfo.bookTypeId << booksInfo.bookDir << booksInfo.bookFileName;
                    
                }
                else
                {
                    qDebug() << "Failed:" << file.fileName() << file.errorString() << endl;
                    return;
                }
            }
        

        So the binary file looks like this.

        0x01 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xcb 0x3b 0x8d
        0x12 0x09 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x41 0x7e 0x68 0x5e 0x52 0x73 0x60
        0x71 0x73 0x41 0x00 0x00 0x00 0x00 0x00 0x00 (hex bookDir and hex bookFileName)

        my program prints out 0x01 then 0x00 etc... never gets the correct data....

        its suppose to be...

        m_binaryVersion = 0x01
        bookCount = 0x02
        bookhash = 0xcb 0x3b 0x8d 0x12
        bookhashId = 0x09
        booktype = 0x41 0x7e 0x68 0x5e 0x52 0x73 0x60 0x71 0x73
        booktypeid = 0x41
        bookDir = hex etc...
        bookFileName = hex etc..

        Also is there a way i can loop thru a qdatastream twice? since the book count is 2 and contains the same structure again?

        Or should i just use readRawData to read it into chunks and parse it from there?

        JonBJ Offline
        JonBJ Offline
        JonB
        wrote on last edited by JonB
        #3

        @Styx
        As @Christian-Ehrlicher has just written QDataStream imposes its own format. Try streaming your BinaryType/Books out << to a file and see what it looks like :)

        1 Reply Last reply
        0
        • StyxS Offline
          StyxS Offline
          Styx
          wrote on last edited by
          #4

          How should i go about reading the data then? Qfile read or qbytearray readall then cover the data from there? Readrawdata?

          A small example would be helpfull?

          jsulmJ JKSHJ 2 Replies Last reply
          0
          • StyxS Styx

            How should i go about reading the data then? Qfile read or qbytearray readall then cover the data from there? Readrawdata?

            A small example would be helpfull?

            jsulmJ Offline
            jsulmJ Offline
            jsulm
            Lifetime Qt Champion
            wrote on last edited by
            #5

            @Styx There is an example in the documentation: https://doc.qt.io/qt-5/qdatastream.html

            QFile file("file.dat");
            file.open(QIODevice::ReadOnly);
            QDataStream in(&file);    // read the data serialized from the file
            QString str;
            qint32 a;
            in >> str >> a;
            

            https://forum.qt.io/topic/113070/qt-code-of-conduct

            1 Reply Last reply
            0
            • Christian EhrlicherC Offline
              Christian EhrlicherC Offline
              Christian Ehrlicher
              Lifetime Qt Champion
              wrote on last edited by
              #6

              @Styx said in Qdatastreams and binary files.:

              Qfile read or qbytearray readall then cover the data from there?

              Yes

              @jsulm: the data is not created with QDataStream, so reading with QDataStream will not (really) work.

              Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
              Visit the Qt Academy at https://academy.qt.io/catalog

              jsulmJ 1 Reply Last reply
              2
              • Christian EhrlicherC Christian Ehrlicher

                @Styx said in Qdatastreams and binary files.:

                Qfile read or qbytearray readall then cover the data from there?

                Yes

                @jsulm: the data is not created with QDataStream, so reading with QDataStream will not (really) work.

                jsulmJ Offline
                jsulmJ Offline
                jsulm
                Lifetime Qt Champion
                wrote on last edited by
                #7

                @Christian-Ehrlicher Ah, OK. I was thinking other way around - it is early morning here :-)

                https://forum.qt.io/topic/113070/qt-code-of-conduct

                1 Reply Last reply
                1
                • StyxS Styx

                  How should i go about reading the data then? Qfile read or qbytearray readall then cover the data from there? Readrawdata?

                  A small example would be helpfull?

                  JKSHJ Offline
                  JKSHJ Offline
                  JKSH
                  Moderators
                  wrote on last edited by JKSH
                  #8

                  @Styx said in Qdatastreams and binary files.:

                  How should i go about reading the data then?

                  First, you need to clarify:

                  1. What is the binary layout of your file?
                  2. Are the numbers in your file stored in little-endian or big-endian format?
                  3. How are your text strings encoded? ASCII? UTF-8? Something else?

                  @Styx said in Qdatastreams and binary files.:

                  So the binary file looks like this.

                  0x01 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0xcb 0x3b 0x8d
                  0x12 ....

                  its suppose to be

                  m_binaryVersion = 0x01
                  bookCount = 0x02
                  bookhash = 0xcb 0x3b 0x8d 0x12

                  OK, so your file layout is something like this...

                  Byte Offset Content
                  0 m_binaryVersion
                  1-4 ???
                  5 bookCount
                  6-12 ????
                  13-16 bookHash

                  There are 4 null bytes between 0x01 and 0x02, and there are 7 null bytes between 0x02 and 0xcb 0x3b 0x8d 0x12. What are these null bytes for?

                  Is bookCount supposed to occupy Bytes 5-12 as a qint64?

                  Qt Doc Search for browsers: forum.qt.io/topic/35616/web-browser-extension-for-improved-doc-searches

                  1 Reply Last reply
                  3
                  • StyxS Offline
                    StyxS Offline
                    Styx
                    wrote on last edited by Styx
                    #9

                    how do i read chunks or sections with qbytearray? Should i store it in one buffer then read the section with qfile read?

                    1-4 bytes are just padding i believe 5 is the ammount of books that is written to the struct

                    0xcb 0x3b 0x8d is the bookhash

                    0x12 is the bookhashId

                    jsulmJ JKSHJ 2 Replies Last reply
                    0
                    • StyxS Styx

                      how do i read chunks or sections with qbytearray? Should i store it in one buffer then read the section with qfile read?

                      1-4 bytes are just padding i believe 5 is the ammount of books that is written to the struct

                      0xcb 0x3b 0x8d is the bookhash

                      0x12 is the bookhashId

                      jsulmJ Offline
                      jsulmJ Offline
                      jsulm
                      Lifetime Qt Champion
                      wrote on last edited by
                      #10

                      @Styx said in Qdatastreams and binary files.:

                      how do i read chunks or sections with qbytearray?

                      https://doc.qt.io/qt-5/qbytearray.html#mid

                      https://forum.qt.io/topic/113070/qt-code-of-conduct

                      1 Reply Last reply
                      3
                      • StyxS Styx

                        how do i read chunks or sections with qbytearray? Should i store it in one buffer then read the section with qfile read?

                        1-4 bytes are just padding i believe 5 is the ammount of books that is written to the struct

                        0xcb 0x3b 0x8d is the bookhash

                        0x12 is the bookhashId

                        JKSHJ Offline
                        JKSHJ Offline
                        JKSH
                        Moderators
                        wrote on last edited by
                        #11

                        @Styx said in Qdatastreams and binary files.:

                        1-4 bytes are just padding

                        OK

                        i believe 5 is the ammount of books that is written to the struct

                        So the maximum number of books is 255?

                        What are bytes 6-12?

                        Qt Doc Search for browsers: forum.qt.io/topic/35616/web-browser-extension-for-improved-doc-searches

                        1 Reply Last reply
                        0
                        • StyxS Offline
                          StyxS Offline
                          Styx
                          wrote on last edited by
                          #12

                          The null bytes are just null padding. max number of books are 5 each book has its own bookhash and bookhashId booktype bookdir and bookfilename

                          Would use qbytearray mid grab the max number of books which is 5 then loop thru the rest of the qbytearray to grab each bookinformation.

                          JKSHJ 1 Reply Last reply
                          0
                          • StyxS Styx

                            The null bytes are just null padding. max number of books are 5 each book has its own bookhash and bookhashId booktype bookdir and bookfilename

                            Would use qbytearray mid grab the max number of books which is 5 then loop thru the rest of the qbytearray to grab each bookinformation.

                            JKSHJ Offline
                            JKSHJ Offline
                            JKSH
                            Moderators
                            wrote on last edited by
                            #13

                            @Styx said in Qdatastreams and binary files.:

                            The null bytes are just null padding. max number of books are 5 each book has its own bookhash and bookhashId booktype bookdir and bookfilename

                            OK

                            Would use qbytearray mid grab the max number of books which is 5 then loop thru the rest of the qbytearray to grab each bookinformation.

                            Sounds good

                            Qt Doc Search for browsers: forum.qt.io/topic/35616/web-browser-extension-for-improved-doc-searches

                            1 Reply Last reply
                            0
                            • StyxS Offline
                              StyxS Offline
                              Styx
                              wrote on last edited by Styx
                              #14

                              is there a way to loop thru the bytearray without using mid?

                              was using...

                              Books bookinfo;
                               bookinfo.bookCount =  (bytearray.at(5) & 0xFFFF);
                              
                              jsulmJ 1 Reply Last reply
                              0
                              • StyxS Styx

                                is there a way to loop thru the bytearray without using mid?

                                was using...

                                Books bookinfo;
                                 bookinfo.bookCount =  (bytearray.at(5) & 0xFFFF);
                                
                                jsulmJ Offline
                                jsulmJ Offline
                                jsulm
                                Lifetime Qt Champion
                                wrote on last edited by
                                #15

                                @Styx said in Qdatastreams and binary files.:

                                is there a way to loop thru the bytearray without using mid?

                                Sure (https://doc.qt.io/qt-5/qbytearray.html):
                                QByteRef operator[](int i)
                                char operator[](int i) const
                                char operator[](uint i) const
                                QByteRef operator[](uint i)

                                So

                                for (int i = 0; i < 3; ++i)
                                    bytearray[i];
                                

                                https://forum.qt.io/topic/113070/qt-code-of-conduct

                                1 Reply Last reply
                                3
                                • StyxS Offline
                                  StyxS Offline
                                  Styx
                                  wrote on last edited by
                                  #16

                                  So once i readall the file into the qbytearray i would use mid to break up the byte offset and copy them to another bytearray?

                                  Is there away to get around not having to use so many qbytearray to copy data?

                                  How would seek and read work from a Qfile?

                                  02 - book count 
                                  
                                  00 00 00 00 00 00 00 - (padding)
                                  
                                  fb 2b 7d 13 - bookhash
                                  
                                  09 - bookhashid
                                  
                                  00 00 00 00 00 00 00 - (padding)
                                  
                                  44 6e 49 4f 43 44 61 62 64 - booktype
                                  
                                  42 - booktypeid
                                  
                                  00 00 00 00 00 00 00 - (padding)
                                  
                                  00 00 00 00 00 00 00 00 00 00 00 00 - bookDir and bookFileName (Qstring)
                                  
                                  00 00 00 00 00 00 00 - (padding)
                                  
                                  1a 10 a2 ae - bookhash
                                  
                                  08 - bookhashid
                                  
                                  00 00 00 00 00 00 00 
                                  
                                  41 6S 64 4f 47 61 49 44 - booktype
                                  
                                  3c - booktypeid
                                  
                                  00 00 00 00 00 00 00 
                                  
                                  00 00 00 00 00 00 00 00 00 00 00 00 - bookDir and bookFileName (Qstring)
                                  

                                  This is a example of a file i am reading i was trying to read it into a struct but not sure if that is the correct method. Qdatastreams cant be used because the file wasn't written by qdatastreams.

                                  as you seen in the code the book count is looped base on the same information provided.

                                  1 Reply Last reply
                                  0
                                  • mrjjM Offline
                                    mrjjM Offline
                                    mrjj
                                    Lifetime Qt Champion
                                    wrote on last edited by
                                    #17

                                    Hi
                                    What app produces the file ?
                                    Its not open source so you could get the actual record definitions ?

                                    1 Reply Last reply
                                    0
                                    • Christian EhrlicherC Offline
                                      Christian EhrlicherC Offline
                                      Christian Ehrlicher
                                      Lifetime Qt Champion
                                      wrote on last edited by
                                      #18

                                      @Styx said in Qdatastreams and binary files.:

                                      Is there away to get around not having to use so many qbytearray to copy data?

                                      Work with a plain const char * pointer

                                      Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                                      Visit the Qt Academy at https://academy.qt.io/catalog

                                      JKSHJ 1 Reply Last reply
                                      2
                                      • Christian EhrlicherC Christian Ehrlicher

                                        @Styx said in Qdatastreams and binary files.:

                                        Is there away to get around not having to use so many qbytearray to copy data?

                                        Work with a plain const char * pointer

                                        JKSHJ Offline
                                        JKSHJ Offline
                                        JKSH
                                        Moderators
                                        wrote on last edited by JKSH
                                        #19

                                        @Christian-Ehrlicher said in Qdatastreams and binary files.:

                                        Work with a plain const char * pointer

                                        To add to @Christian-Ehrlicher's point: CallQByteArray:data() or QByteArray::constData() to get a raw pointer to your data. Then, you can use pointer arithmetic to extract your data.

                                        QByteArray ba = file.readAll();
                                        const char* data = ba.constData();
                                        
                                        // Assuming that your file is little-endian...
                                        memcpy(&m_binaryVersion, data +  0, sizeof(quint8 ));
                                        memcpy(&bookCount,       data +  5, sizeof(quint8 ));
                                        memcpy(&bookHash,        data + 13, sizeof(quint32));
                                        

                                        EDIT: Code above changed from reinterpret_cast<> to memcpy() for cross-platform safety

                                        Qt Doc Search for browsers: forum.qt.io/topic/35616/web-browser-extension-for-improved-doc-searches

                                        JonBJ 1 Reply Last reply
                                        5
                                        • StyxS Offline
                                          StyxS Offline
                                          Styx
                                          wrote on last edited by
                                          #20

                                          @JKSH Since .data() is null terminated. Think it would be better to use shift left.

                                          // Assuming that your file is little-endian...
                                          m_binaryVersion = *reinterpret_cast<const quint8* >(data +  0 ) >> 8;
                                          bookCount       = *reinterpret_cast<const quint8* >(data +  5) >> 12;
                                          bookHash        = *reinterpret_cast<const quint32*>(data + 13) >> 16;
                                          
                                          // example
                                          bookCount=256 the first byte is '\0' then all the rest will be undetermined.
                                          

                                          Shouldn't have issues calling the index and then looping thru the qbytearray to print out the data as well.

                                          JKSHJ JonBJ 2 Replies Last reply
                                          0

                                          • Login

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Users
                                          • Groups
                                          • Search
                                          • Get Qt Extensions
                                          • Unsolved