Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. How to write entire QVector to a binary file?
Forum Updated to NodeBB v4.3 + New Features

How to write entire QVector to a binary file?

Scheduled Pinned Locked Moved Unsolved General and Desktop
qvectorbinary formatqdatastream
46 Posts 8 Posters 13.1k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • Christian EhrlicherC Offline
    Christian EhrlicherC Offline
    Christian Ehrlicher
    Lifetime Qt Champion
    wrote on last edited by
    #11

    @CJha said in How to write entire QVector to a binary file?:

    and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

    Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

    Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
    Visit the Qt Academy at https://academy.qt.io/catalog

    CJhaC 1 Reply Last reply
    1
    • CJhaC CJha

      @Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.

      VRoninV Offline
      VRoninV Offline
      VRonin
      wrote on last edited by
      #12

      @CJha said in How to write entire QVector to a binary file?:

      I have to read it in Matlab.

      If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

      QFile file("BinTest.bin");
      file.open(QIODevice::WriteOnly | QIODevice::Text);
      QTextStream out(&file);
      QVector<double> vec;
      for(int ii = 0; ii < 10; ++ii){
          vec << ii * 0.33;
      }
      for(auto&& val : vec)
      out << val << ',';
      file.close();
      

      "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
      ~Napoleon Bonaparte

      On a crusade to banish setIndexWidget() from the holy land of Qt

      CJhaC 1 Reply Last reply
      1
      • Christian EhrlicherC Christian Ehrlicher

        @CJha said in How to write entire QVector to a binary file?:

        and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

        Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?

        CJhaC Offline
        CJhaC Offline
        CJha
        wrote on last edited by
        #13

        @Christian-Ehrlicher I didn't know that before I saw the source code for QDataStream. I assumed since for a QVector data points are in adjacent memory positions pushing an entire vector into a binary file must be faster than iterating over it.

        1 Reply Last reply
        0
        • CJhaC CJha

          @J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
          @Christian-Ehrlicher @JonB The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.

          JonBJ Offline
          JonBJ Offline
          JonB
          wrote on last edited by JonB
          #14

          @CJha said in How to write entire QVector to a binary file?:

          The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points

          Yep. That's what has to be done, and it's what the << serializer does, as @J-Hilk showed you. The only other way would be if you can get the address of contiguous QVector memory and save from there, which I'm guessing can be done. However...

          ...If @VRonin's latest post is correct and you're supposed to produce text instead to export then you cannot help but do it one-by-one....

          P.S.
          BTW, you'd have to test, but my guess is that code to output data points one-by-one instead of in a contiguous clump is not what would be slow over 1,000,000 points --- rather, the size of the output written to file will be what is significant....

          1 Reply Last reply
          0
          • Christian EhrlicherC Offline
            Christian EhrlicherC Offline
            Christian Ehrlicher
            Lifetime Qt Champion
            wrote on last edited by
            #15

            You're correct with the adjacent memory but QVector can also hold objects so memcpy'ing it out will not work there. You can use memcpy if you want in your case but QDataStream is generic and has no optimizations for such things.

            Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
            Visit the Qt Academy at https://academy.qt.io/catalog

            1 Reply Last reply
            3
            • VRoninV VRonin

              @CJha said in How to write entire QVector to a binary file?:

              I have to read it in Matlab.

              If I remeber correctly Matlab reads csv (aka text) so QTextStream is what you want. something like:

              QFile file("BinTest.bin");
              file.open(QIODevice::WriteOnly | QIODevice::Text);
              QTextStream out(&file);
              QVector<double> vec;
              for(int ii = 0; ii < 10; ++ii){
                  vec << ii * 0.33;
              }
              for(auto&& val : vec)
              out << val << ',';
              file.close();
              
              CJhaC Offline
              CJhaC Offline
              CJha
              wrote on last edited by
              #16

              @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

              JonBJ jsulmJ J.HilkJ 3 Replies Last reply
              0
              • CJhaC CJha

                @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                JonBJ Offline
                JonBJ Offline
                JonB
                wrote on last edited by
                #17

                @CJha
                OK, so STOP and tell us exactly what format MatLab will accept, and take it from there...

                1 Reply Last reply
                2
                • CJhaC CJha

                  @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                  jsulmJ Offline
                  jsulmJ Offline
                  jsulm
                  Lifetime Qt Champion
                  wrote on last edited by
                  #18

                  @CJha But does Matlab support any binary format? You have to write in a format supported by Matlab.

                  https://forum.qt.io/topic/113070/qt-code-of-conduct

                  1 Reply Last reply
                  0
                  • CJhaC CJha

                    @VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.

                    J.HilkJ Offline
                    J.HilkJ Offline
                    J.Hilk
                    Moderators
                    wrote on last edited by
                    #19

                    @CJha said in How to write entire QVector to a binary file?:

                    I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

                    actually stop right here!

                    if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                    I'm sure there's an other - in memory - way to hand over those data points


                    Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


                    Q: What's that?
                    A: It's blue light.
                    Q: What does it do?
                    A: It turns blue.

                    CJhaC 1 Reply Last reply
                    1
                    • CJhaC Offline
                      CJhaC Offline
                      CJha
                      wrote on last edited by
                      #20

                      @JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on

                      jsulmJ 1 Reply Last reply
                      0
                      • CJhaC CJha

                        @JonB @jsulm Matlab supports the binary format. There are functions such as fopen, fread, fseek etc. to read and write binary files. I have read binary file written using QDataStream in Matlab using fread and so on

                        jsulmJ Offline
                        jsulmJ Offline
                        jsulm
                        Lifetime Qt Champion
                        wrote on last edited by
                        #21

                        @CJha said in How to write entire QVector to a binary file?:

                        Matlab supports the binary format

                        Do you have its specification?

                        https://forum.qt.io/topic/113070/qt-code-of-conduct

                        CJhaC 1 Reply Last reply
                        0
                        • J.HilkJ J.Hilk

                          @CJha said in How to write entire QVector to a binary file?:

                          I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours

                          actually stop right here!

                          if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                          I'm sure there's an other - in memory - way to hand over those data points

                          CJhaC Offline
                          CJhaC Offline
                          CJha
                          wrote on last edited by
                          #22

                          @J-Hilk I am not sure what you mean by

                          if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                          Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                          JonBJ J.HilkJ 2 Replies Last reply
                          0
                          • CJhaC CJha

                            @J-Hilk I am not sure what you mean by

                            if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                            Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                            JonBJ Offline
                            JonBJ Offline
                            JonB
                            wrote on last edited by JonB
                            #23

                            @CJha
                            You may (well) know more than I, but can MatLab read and process 8MB of new data per second, at the same time as something else is producing it? And, separately, do you really generate 1 million new data points per second?

                            Also, as @J-Hilk said, wouldn't sending a pipe stream (e.g. a socket?) be better than writing to file and reading back? Does Matlab accept incoming data elsewhere than in a file?

                            CJhaC 1 Reply Last reply
                            0
                            • jsulmJ jsulm

                              @CJha said in How to write entire QVector to a binary file?:

                              Matlab supports the binary format

                              Do you have its specification?

                              CJhaC Offline
                              CJhaC Offline
                              CJha
                              wrote on last edited by
                              #24

                              @jsulm It is highly compatible. Here are the links for fopen, fread, fseek. I all these I can specify the format, ByteOrder, size of data (such as int, double, etc), and quite a few other things.
                              I don't think Matlab is the restrictive thing here, I can read any type of binary file in Matlab as long as I know how it is written.

                              JonBJ 1 Reply Last reply
                              0
                              • CJhaC CJha

                                @jsulm It is highly compatible. Here are the links for fopen, fread, fseek. I all these I can specify the format, ByteOrder, size of data (such as int, double, etc), and quite a few other things.
                                I don't think Matlab is the restrictive thing here, I can read any type of binary file in Matlab as long as I know how it is written.

                                JonBJ Offline
                                JonBJ Offline
                                JonB
                                wrote on last edited by JonB
                                #25

                                @CJha
                                The code to write a QVector to file in the way you want, as fast as possible in one blob not one-by-one, is given in e.g. https://www.qtcentre.org/threads/65713-Output-a-QVector-of-doubles-to-a-binary-file-(without-using-QDatastream)?p=289540#post289540 :

                                qint64 bytesWritten = file.write(reinterpret_cast<const char*>(vec.constData()), sizeof(double) * vec.size());
                                

                                EDIT I think you will want reinterpret_cast<> rather than static_cast<> here as shown in that post, so I have altered the code line to use that.

                                CJhaC 1 Reply Last reply
                                4
                                • JonBJ JonB

                                  @CJha
                                  You may (well) know more than I, but can MatLab read and process 8MB of new data per second, at the same time as something else is producing it? And, separately, do you really generate 1 million new data points per second?

                                  Also, as @J-Hilk said, wouldn't sending a pipe stream (e.g. a socket?) be better than writing to file and reading back? Does Matlab accept incoming data elsewhere than in a file?

                                  CJhaC Offline
                                  CJhaC Offline
                                  CJha
                                  wrote on last edited by
                                  #26

                                  @JonB No, Matlab is going to read it at a later time. When data is being generated it is just stored in a binary file for later use by Matlab. And yes, Matlab is slower but it doesn't matter if it takes 1 second or 1 day to read the file as the researchers can just start loading the file in the night and come back in the morning to work on it (many researchers wait for times like 24 to 36 hours for files to get processed).

                                  And yes, I am generating data at 1 million doubles per second. I am using National Instruments and Measurement Computing DAQ boards, controlling both through Qt and C++ and these boards are capable of generating 1 million doubles per second.

                                  1 Reply Last reply
                                  1
                                  • CJhaC CJha

                                    @J-Hilk I am not sure what you mean by

                                    if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                    Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                                    J.HilkJ Offline
                                    J.HilkJ Offline
                                    J.Hilk
                                    Moderators
                                    wrote on last edited by J.Hilk
                                    #27

                                    @CJha said in How to write entire QVector to a binary file?:

                                    @J-Hilk I am not sure what you mean by

                                    if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                    Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                                    its not about the speed, its about the amount of times written into the cell, Samsung for examples says their ssd's are "built to handle 150 terabytes written" with, lets say 1 million points of double (8 bytes each) per second would mean your ssd is done for in roughly 200 days, instead of the approximated 10 years.

                                    also you have to coordinate read and write access of the file, so that Matlab and your Qt Programm to not try to access the file at the same time with potential data loss etc


                                    Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


                                    Q: What's that?
                                    A: It's blue light.
                                    Q: What does it do?
                                    A: It turns blue.

                                    CJhaC 1 Reply Last reply
                                    2
                                    • JonBJ JonB

                                      @CJha
                                      The code to write a QVector to file in the way you want, as fast as possible in one blob not one-by-one, is given in e.g. https://www.qtcentre.org/threads/65713-Output-a-QVector-of-doubles-to-a-binary-file-(without-using-QDatastream)?p=289540#post289540 :

                                      qint64 bytesWritten = file.write(reinterpret_cast<const char*>(vec.constData()), sizeof(double) * vec.size());
                                      

                                      EDIT I think you will want reinterpret_cast<> rather than static_cast<> here as shown in that post, so I have altered the code line to use that.

                                      CJhaC Offline
                                      CJhaC Offline
                                      CJha
                                      wrote on last edited by
                                      #28

                                      @JonB Thanks, this might just work for me. I will try it out now and see what's the difference in speed of writing using this method and iterative method and will post it back here.

                                      1 Reply Last reply
                                      1
                                      • J.HilkJ J.Hilk

                                        @CJha said in How to write entire QVector to a binary file?:

                                        @J-Hilk I am not sure what you mean by

                                        if this program is used more than once, you're going to destroy your HD/SSD very quickly!

                                        Given that 1 million doubles are 8 million bytes, I think modern processors and disk drives can handle such speed easily.

                                        its not about the speed, its about the amount of times written into the cell, Samsung for examples says their ssd's are "built to handle 150 terabytes written" with, lets say 1 million points of double (8 bytes each) per second would mean your ssd is done for in roughly 200 days, instead of the approximated 10 years.

                                        also you have to coordinate read and write access of the file, so that Matlab and your Qt Programm to not try to access the file at the same time with potential data loss etc

                                        CJhaC Offline
                                        CJhaC Offline
                                        CJha
                                        wrote on last edited by
                                        #29

                                        @J-Hilk That's a good point, but it's not the case for me as the data write and data read happens at different times. Also, SSD lifetime doesn't matter as these researchers have lots of funding and SSD is a cheap item for them. My job is to give them what they ask for, and if they ruin their SSD in 200 days that is up to them (of course I will tell them that it can ruin their SSD fast but that's all I can do).

                                        J.HilkJ JonBJ 3 Replies Last reply
                                        1
                                        • CJhaC CJha

                                          @J-Hilk That's a good point, but it's not the case for me as the data write and data read happens at different times. Also, SSD lifetime doesn't matter as these researchers have lots of funding and SSD is a cheap item for them. My job is to give them what they ask for, and if they ruin their SSD in 200 days that is up to them (of course I will tell them that it can ruin their SSD fast but that's all I can do).

                                          J.HilkJ Offline
                                          J.HilkJ Offline
                                          J.Hilk
                                          Moderators
                                          wrote on last edited by
                                          #30

                                          @CJha Fair enough,
                                          The customer gets, what the customer wants.

                                          I can't remember the amount of times I had to do implement stuff I whole heartily disagreed with ...🙈


                                          Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


                                          Q: What's that?
                                          A: It's blue light.
                                          Q: What does it do?
                                          A: It turns blue.

                                          1 Reply Last reply
                                          1

                                          • Login

                                          • Login or register to search.
                                          • First post
                                            Last post
                                          0
                                          • Categories
                                          • Recent
                                          • Tags
                                          • Popular
                                          • Users
                                          • Groups
                                          • Search
                                          • Get Qt Extensions
                                          • Unsolved