Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. How to write entire QVector to a binary file?
Forum Updated to NodeBB v4.3 + New Features

How to write entire QVector to a binary file?

Scheduled Pinned Locked Moved Unsolved General and Desktop
qvectorbinary formatqdatastream
46 Posts 8 Posters 13.1k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • Christian EhrlicherC Offline
    Christian EhrlicherC Offline
    Christian Ehrlicher
    Lifetime Qt Champion
    wrote on last edited by
    #4

    @CJha said in How to write entire QVector to a binary file?:

    so that I have an idea how to read my binary file?

    Read it with QDataStream. The format is internal.

    Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
    Visit the Qt Academy at https://academy.qt.io/catalog

    CJhaC 1 Reply Last reply
    1
    • Christian EhrlicherC Christian Ehrlicher

      @CJha said in How to write entire QVector to a binary file?:

      so that I have an idea how to read my binary file?

      Read it with QDataStream. The format is internal.

      CJhaC Offline
      CJhaC Offline
      CJha
      wrote on last edited by
      #5

      @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

      JonBJ VRoninV 2 Replies Last reply
      0
      • Christian EhrlicherC Offline
        Christian EhrlicherC Offline
        Christian Ehrlicher
        Lifetime Qt Champion
        wrote on last edited by
        #6

        @CJha said in How to write entire QVector to a binary file?:

        I have to read it in Matlab.

        Then create your own format or check what matlab needs to read it with it's built-in functions.

        Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
        Visit the Qt Academy at https://academy.qt.io/catalog

        1 Reply Last reply
        2
        • CJhaC CJha

          @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

          JonBJ Offline
          JonBJ Offline
          JonB
          wrote on last edited by JonB
          #7

          @CJha
          Then do not use a Qt serialization operator! Use whatever is going to be acceptable at the MatLab side, e.g. just a plain array's-worth of the data, or whatever. Qt serialization is for Qt programs!

          1 Reply Last reply
          0
          • CJhaC CJha

            @J-Hilk I am unable to find any examples to tell me how to get the header. The QDataStream class description only says just a little about serializing QVectors do you know any examples that I can follow so that I have an idea how to read my binary file?

            J.HilkJ Offline
            J.HilkJ Offline
            J.Hilk
            Moderators
            wrote on last edited by J.Hilk
            #8

            @CJha you can take a look into the source code

            no idea what version you're using, but I found this

            00717b7a-3822-470e-a26a-220eb197ba6e-image.png

            066e8007-1c4a-4394-aec7-7138631cfe9e-image.png

            link


            Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


            Q: What's that?
            A: It's blue light.
            Q: What does it do?
            A: It turns blue.

            Christian EhrlicherC 1 Reply Last reply
            0
            • J.HilkJ J.Hilk

              @CJha you can take a look into the source code

              no idea what version you're using, but I found this

              00717b7a-3822-470e-a26a-220eb197ba6e-image.png

              066e8007-1c4a-4394-aec7-7138631cfe9e-image.png

              link

              Christian EhrlicherC Offline
              Christian EhrlicherC Offline
              Christian Ehrlicher
              Lifetime Qt Champion
              wrote on last edited by
              #9

              @J-Hilk But this is internal and may change (although the chance is very very low)

              Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
              Visit the Qt Academy at https://academy.qt.io/catalog

              1 Reply Last reply
              1
              • CJhaC Offline
                CJhaC Offline
                CJha
                wrote on last edited by
                #10

                @J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
                @Christian-Ehrlicher @JonB The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                JonBJ 1 Reply Last reply
                0
                • Christian EhrlicherC Offline
                  Christian EhrlicherC Offline
                  Christian Ehrlicher
                  Lifetime Qt Champion
                  wrote on last edited by
                  #11

                  @CJha said in How to write entire QVector to a binary file?:

                  and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                  Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

                  Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                  Visit the Qt Academy at https://academy.qt.io/catalog

                  CJhaC 1 Reply Last reply
                  1
                  • CJhaC CJha

                    @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

                    VRoninV Offline
                    VRoninV Offline
                    VRonin
                    wrote on last edited by
                    #12

                    @CJha said in How to write entire QVector to a binary file?:

                    I have to read it in Matlab.

                    If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

                    QFile file("BinTest.bin");
                    file.open(QIODevice::WriteOnly | QIODevice::Text);
                    QTextStream out(&file);
                    QVector<double> vec;
                    for(int ii = 0; ii < 10; ++ii){
                        vec << ii * 0.33;
                    }
                    for(auto&& val : vec)
                    out << val << ',';
                    file.close();
                    

                    "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
                    ~Napoleon Bonaparte

                    On a crusade to banish setIndexWidget() from the holy land of Qt

                    CJhaC 1 Reply Last reply
                    1
                    • Christian EhrlicherC Christian Ehrlicher

                      @CJha said in How to write entire QVector to a binary file?:

                      and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                      Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

                      CJhaC Offline
                      CJhaC Offline
                      CJha
                      wrote on last edited by
                      #13

                      @Christian-Ehrlicher I didn't know that before I saw the source code for QDataStream. I assumed since for a QVector data points are in adjacent memory positions pushing an entire vector into a binary file must be faster than iterating over it.

                      1 Reply Last reply
                      0
                      • CJhaC CJha

                        @J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
                        @Christian-Ehrlicher @JonB The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

                        JonBJ Offline
                        JonBJ Offline
                        JonB
                        wrote on last edited by JonB
                        #14

                        @CJha said in How to write entire QVector to a binary file?:

                        The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points

                        Yep. That's what has to be done, and it's what the << serializer does, as @J-Hilk showed you. The only other way would be if you can get the address of contiguous QVector memory and save from there, which I'm guessing can be done. However...

                        ...If @VRonin's latest post is correct and you're supposed to produce text instead to export then you cannot help but do it one-by-one....

                        P.S.
                        BTW, you'd have to test, but my guess is that code to output data points one-by-one instead of in a contiguous clump is not what would be slow over 1,000,000 points --- rather, the size of the output written to file will be what is significant....

                        1 Reply Last reply
                        0
                        • Christian EhrlicherC Offline
                          Christian EhrlicherC Offline
                          Christian Ehrlicher
                          Lifetime Qt Champion
                          wrote on last edited by
                          #15

                          You're correct with the adjacent memory but QVector can also hold objects so memcpy'ing it out will not work there. You can use memcpy if you want in your case but QDataStream is generic and has no optimizations for such things.

                          Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
                          Visit the Qt Academy at https://academy.qt.io/catalog

                          1 Reply Last reply
                          3
                          • VRoninV VRonin

                            @CJha said in How to write entire QVector to a binary file?:

                            I have to read it in Matlab.

                            If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

                            QFile file("BinTest.bin");
                            file.open(QIODevice::WriteOnly | QIODevice::Text);
                            QTextStream out(&file);
                            QVector<double> vec;
                            for(int ii = 0; ii < 10; ++ii){
                                vec << ii * 0.33;
                            }
                            for(auto&& val : vec)
                            out << val << ',';
                            file.close();
                            
                            CJhaC Offline
                            CJhaC Offline
                            CJha
                            wrote on last edited by
                            #16

                            @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                            JonBJ jsulmJ J.HilkJ 3 Replies Last reply
                            0
                            • CJhaC CJha

                              @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                              JonBJ Offline
                              JonBJ Offline
                              JonB
                              wrote on last edited by
                              #17

                              @CJha
                              OK, so STOP and tell us exactly what format MatLab will accept, and take it from there...

                              1 Reply Last reply
                              2
                              • CJhaC CJha

                                @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                                jsulmJ Offline
                                jsulmJ Offline
                                jsulm
                                Lifetime Qt Champion
                                wrote on last edited by
                                #18

                                @CJha But does Matlab support any binary format? You have to write in a format supported by Matlab.

                                https://forum.qt.io/topic/113070/qt-code-of-conduct

                                1 Reply Last reply
                                0
                                • CJhaC CJha

                                  @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                                  J.HilkJ Offline
                                  J.HilkJ Offline
                                  J.Hilk
                                  Moderators
                                  wrote on last edited by
                                  #19

                                  @CJha said in How to write entire QVector to a binary file?:

                                  I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

                                  actually stop right here!

                                  if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                  I'm sure there's an other - in memory - way to hand over those data points


                                  Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


                                  Q: What's that?
                                  A: It's blue light.
                                  Q: What does it do?
                                  A: It turns blue.

                                  CJhaC 1 Reply Last reply
                                  1
                                  • CJhaC Offline
                                    CJhaC Offline
                                    CJha
                                    wrote on last edited by
                                    #20

                                    @JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on

                                    jsulmJ 1 Reply Last reply
                                    0
                                    • CJhaC CJha

                                      @JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on

                                      jsulmJ Offline
                                      jsulmJ Offline
                                      jsulm
                                      Lifetime Qt Champion
                                      wrote on last edited by
                                      #21

                                      @CJha said in How to write entire QVector to a binary file?:

                                      Matlab supports the binary format

                                      Do you have its specification?

                                      https://forum.qt.io/topic/113070/qt-code-of-conduct

                                      CJhaC 1 Reply Last reply
                                      0
                                      • J.HilkJ J.Hilk

                                        @CJha said in How to write entire QVector to a binary file?:

                                        I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

                                        actually stop right here!

                                        if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                        I'm sure there's an other - in memory - way to hand over those data points

                                        CJhaC Offline
                                        CJhaC Offline
                                        CJha
                                        wrote on last edited by
                                        #22

                                        @J-Hilk I am not sure what you mean by

                                        if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                        Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                                        JonBJ J.HilkJ 2 Replies Last reply
                                        0
                                        • CJhaC CJha

                                          @J-Hilk I am not sure what you mean by

                                          if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                          Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                                          JonBJ Offline
                                          JonBJ Offline
                                          JonB
                                          wrote on last edited by JonB
                                          #23

                                          @CJha
                                          You may (well) know more than I, but can MatLab read and process 8MB of new data per second, at the same time as something else is producing it? And, separately, do you really generate 1 million new data points per second?

                                          Also, as @J-Hilk said, wouldn't sending a pipe stream (e.g. a socket?) be better than writing to file and reading back? Does Matlab accept incoming data elsewhere than in a file?

                                          CJhaC 1 Reply Last reply
                                          0

                                          • Login

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Users
                                          • Groups
                                          • Search
                                          • Get Qt Extensions
                                          • Unsolved