Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. QDataStream bad_alloc corrupt file
Forum Updated to NodeBB v4.3 + New Features

QDataStream bad_alloc corrupt file

Scheduled Pinned Locked Moved Unsolved General and Desktop
16 Posts 4 Posters 1.3k Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M Offline
    M Offline
    maxp
    wrote on last edited by
    #1

    Hi, when opening a file with QFile (QIODevice::ReadOnly) and using QDataStream to read in the contents of a file that I wrote with QFile and QDataStream everything works as it should. But when I delete a part of the content and try to read it I get a std::bad_alloc. How should I use QDataStream to be robust against corrupt files? I cant find an example that does this. Also using status QDataStream::ReadPastEnd and QDataStream::ReadCorruptData don't appear to help me. The status is always QDataStream::Ok.

    So my question is, can someone show me an example of how to use QDataStream to be robust against reading files containing adapted data.

    struct Data
    {
        uint32_t port;
        QHostAddress ip;
        QString category;
    };
    QDataStream& operator<<(QDataStream& stream, Data const& data)
    {
        stream << data.port << data.ip << data.category;
        return stream;
    }
    QDataStream& operator>>(QDataStream& stream, Data& data) {
        stream >> data.port;
        stream >> data.ip;
        stream >> data.category;
        return stream;
    }
    
    QList<Data> data;
    
    QFile file("data.txt");
    if (!file.open(QIODevice::ReadOnly)) { return; }
    QDataStream stream(&file);
    stream.setVersion(QDataStream::Qt_5_12);
    stream >> data; // std::bad_alloc happens here
    file.close();
    
    JonBJ 1 Reply Last reply
    0
    • Christian EhrlicherC Offline
      Christian EhrlicherC Offline
      Christian Ehrlicher
      Lifetime Qt Champion
      wrote on last edited by
      #2

      And how do you write it out to data.txt (why txt - it's a binary file, no text)

      Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
      Visit the Qt Academy at https://academy.qt.io/catalog

      1 Reply Last reply
      0
      • M Offline
        M Offline
        maxp
        wrote on last edited by
        #3

        Ah yes, actually it was "data.ini", but I have made a small example here and the fact that it has extension txt does not really change the working right?

        Christian EhrlicherC 1 Reply Last reply
        0
        • M maxp

          Ah yes, actually it was "data.ini", but I have made a small example here and the fact that it has extension txt does not really change the working right?

          Christian EhrlicherC Offline
          Christian EhrlicherC Offline
          Christian Ehrlicher
          Lifetime Qt Champion
          wrote on last edited by
          #4

          @maxp said in QDataStream bad_alloc corrupt file:

          it has extension txt does not really change the working right?

          Correct, but as I already said the part on how you save the file is missing.

          Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
          Visit the Qt Academy at https://academy.qt.io/catalog

          1 Reply Last reply
          0
          • M Offline
            M Offline
            maxp
            wrote on last edited by
            #5

            Sorry, I was trying to immediately submit the second text, but I have to wait 600 seconds apparently :(.
            I write like this to the file, but reading that is no problem. It is only a problem when I open it for example with notepad and delete a part such that the content is corrupt. I want to be able to detect that instead of having the application throw a bad alloc exception or should I just try catch this and is that robust enough?

            // Write to devices file
            QFile file("data.ini");
            if (!file.open(QIODevice::WriteOnly)) return;
            
            // Small data example
            Data data{.port=3300, .ip=QHostAddress(), .category="text"};
            QList<DeviceData> data_list;
            data_list.append(data);
            
            QDataStream stream(&file);
            stream.setVersion(QDataStream::Qt_5_12);
            stream << data_list;
            file.close();
            
            KroMignonK 1 Reply Last reply
            0
            • Christian EhrlicherC Offline
              Christian EhrlicherC Offline
              Christian Ehrlicher
              Lifetime Qt Champion
              wrote on last edited by
              #6

              You must not edit a file generated with QDataStream with a text editor. Either add a try/catch block or add a checksum to your file which can be used to check the integrity

              Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
              Visit the Qt Academy at https://academy.qt.io/catalog

              1 Reply Last reply
              2
              • M maxp

                Sorry, I was trying to immediately submit the second text, but I have to wait 600 seconds apparently :(.
                I write like this to the file, but reading that is no problem. It is only a problem when I open it for example with notepad and delete a part such that the content is corrupt. I want to be able to detect that instead of having the application throw a bad alloc exception or should I just try catch this and is that robust enough?

                // Write to devices file
                QFile file("data.ini");
                if (!file.open(QIODevice::WriteOnly)) return;
                
                // Small data example
                Data data{.port=3300, .ip=QHostAddress(), .category="text"};
                QList<DeviceData> data_list;
                data_list.append(data);
                
                QDataStream stream(&file);
                stream.setVersion(QDataStream::Qt_5_12);
                stream << data_list;
                file.close();
                
                KroMignonK Offline
                KroMignonK Offline
                KroMignon
                wrote on last edited by
                #7

                @maxp said in QDataStream bad_alloc corrupt file:

                It is only a problem when I open it for example with notepad and delete a part such that the content is corrupt

                Yes of course this is a problem!
                Qt is open source, so you could look yourself in QDataStream to find out why this is not possible.
                When you are serializing an QList, then QDataStream firs writes the QList size, and then each item in the list.

                So hacking the output with a text editor will generate a corrupted/unreadable file.

                What did you expect?

                It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth. (Sherlock Holmes)

                1 Reply Last reply
                0
                • M Offline
                  M Offline
                  maxp
                  wrote on last edited by
                  #8

                  Yes indeed, but I just want to make my program not crash in case I have a corrupt file, or a user manipulated it.

                  KroMignonK 1 Reply Last reply
                  0
                  • M Offline
                    M Offline
                    maxp
                    wrote on last edited by
                    #9

                    So at least the try catch / checksum should be added as Christian said.

                    I was wondering why the status was not QDataStream::ReadCorruptData in this case and when it is the case. But I can check it further in the open source code than.

                    1 Reply Last reply
                    0
                    • M maxp

                      Yes indeed, but I just want to make my program not crash in case I have a corrupt file, or a user manipulated it.

                      KroMignonK Offline
                      KroMignonK Offline
                      KroMignon
                      wrote on last edited by KroMignon
                      #10

                      @maxp said in QDataStream bad_alloc corrupt file:

                      Yes indeed, but I just want to make my program not crash in case I have a corrupt file, or a user manipulated it.

                      Then you have to deal with QDataStream::startTransaction() and QDataStream::commitTransaction().
                      For example:

                      QList<DeviceData> data_list;
                      QDataStream stream(&file);
                      stream.setVersion(QDataStream::Qt_5_12);
                      stream.startTransaction();
                      stream >> data_list;
                      if(!stream.commitTransaction())
                      {
                         qDebug() << "ERROR!!!";
                      }
                      

                      It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth. (Sherlock Holmes)

                      1 Reply Last reply
                      0
                      • M Offline
                        M Offline
                        maxp
                        wrote on last edited by
                        #11

                        Thanks for the information, I'm going to look further into the transaction.

                        1 Reply Last reply
                        0
                        • Christian EhrlicherC Offline
                          Christian EhrlicherC Offline
                          Christian Ehrlicher
                          Lifetime Qt Champion
                          wrote on last edited by
                          #12

                          I would guess due to the modification with the text editor the length indicator of the QList is a very large number which results in a bad_alloc. Therefore the stream can not return with ReadCorruptData

                          Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                          Visit the Qt Academy at https://academy.qt.io/catalog

                          1 Reply Last reply
                          2
                          • M maxp

                            Hi, when opening a file with QFile (QIODevice::ReadOnly) and using QDataStream to read in the contents of a file that I wrote with QFile and QDataStream everything works as it should. But when I delete a part of the content and try to read it I get a std::bad_alloc. How should I use QDataStream to be robust against corrupt files? I cant find an example that does this. Also using status QDataStream::ReadPastEnd and QDataStream::ReadCorruptData don't appear to help me. The status is always QDataStream::Ok.

                            So my question is, can someone show me an example of how to use QDataStream to be robust against reading files containing adapted data.

                            struct Data
                            {
                                uint32_t port;
                                QHostAddress ip;
                                QString category;
                            };
                            QDataStream& operator<<(QDataStream& stream, Data const& data)
                            {
                                stream << data.port << data.ip << data.category;
                                return stream;
                            }
                            QDataStream& operator>>(QDataStream& stream, Data& data) {
                                stream >> data.port;
                                stream >> data.ip;
                                stream >> data.category;
                                return stream;
                            }
                            
                            QList<Data> data;
                            
                            QFile file("data.txt");
                            if (!file.open(QIODevice::ReadOnly)) { return; }
                            QDataStream stream(&file);
                            stream.setVersion(QDataStream::Qt_5_12);
                            stream >> data; // std::bad_alloc happens here
                            file.close();
                            
                            JonBJ Offline
                            JonBJ Offline
                            JonB
                            wrote on last edited by JonB
                            #13

                            @maxp
                            As the others have said for your immediate issue.

                            At a conceptual level: I don't know if this is just a "one-off" where you are somehow wanting to recover a corrupt file. But if you have any intention of wanting to be able to edit these files, for recovery or other purposes, think about changing over to a non-binary, text-friendly format like JSON instead of QDataStream.

                            1 Reply Last reply
                            0
                            • M Offline
                              M Offline
                              maxp
                              wrote on last edited by
                              #14

                              Well, recovering the file is not necessary and it does not have to be editable from outside the program.
                              My initial plan was to read the file, not crash and show an error message/log if there is some kind of problem and just carry on with the program.
                              Then it became: try to read the file, catch the bad_alloc to not crash and warn the user to check the file or delete it. (whatever i change in the file seems to lead to the bad_alloc)
                              I shall have to dig deeper in this problem and how the QDataStream/transaction/etc works :).

                              JonBJ 1 Reply Last reply
                              0
                              • M maxp

                                Well, recovering the file is not necessary and it does not have to be editable from outside the program.
                                My initial plan was to read the file, not crash and show an error message/log if there is some kind of problem and just carry on with the program.
                                Then it became: try to read the file, catch the bad_alloc to not crash and warn the user to check the file or delete it. (whatever i change in the file seems to lead to the bad_alloc)
                                I shall have to dig deeper in this problem and how the QDataStream/transaction/etc works :).

                                JonBJ Offline
                                JonBJ Offline
                                JonB
                                wrote on last edited by JonB
                                #15

                                @maxp said in QDataStream bad_alloc corrupt file:

                                bad_alloc

                                Read, say, https://stackoverflow.com/questions/9456728/how-to-deal-with-bad-alloc-in-c for a typical discussion. You can catch it, it's potentially fraught, safest is to exit afterward. Up to you.

                                It's just as possible that a different piece of "bad" data will cause some quite different error.

                                But in general expecting a program to deal satisfactorily with a binary data stream which is simply "incorrect" and not "hang/crash" is pretty optimistic. You're not going to be able to cover all cases as you would like.

                                1 Reply Last reply
                                0
                                • M Offline
                                  M Offline
                                  maxp
                                  wrote on last edited by
                                  #16

                                  Ok thanks, so exiting after the caught bad_alloc is already good. For this program that is enough. But when I would want to read a file in a program which should not just quit after reading a corrupt one I should use something else than.

                                  1 Reply Last reply
                                  0

                                  • Login

                                  • Login or register to search.
                                  • First post
                                    Last post
                                  0
                                  • Categories
                                  • Recent
                                  • Tags
                                  • Popular
                                  • Users
                                  • Groups
                                  • Search
                                  • Get Qt Extensions
                                  • Unsolved