How to write entire QVector to a binary file?
-
Hi, I am trying to write a
QVector<double>
to a binary file usingQDataStream
. According to Qt documentationReading and Writing Qt Collection Classes
The Qt container classes can also be serialized to a QDataStream. These include QList, QLinkedList, QVector, QSet, QHash, and QMap. The stream operators are declared as non-members of the classes.When I do something like this:
QFile file("BinTest.bin"); file.open(QIODevice::WriteOnly); QDataStream out(&file); QVector<double> vec; for(int ii = 0; ii < 10; ++ii){ vec << ii * 0.33; } out << vec; file.close();
The output contains garbage. I tried serializing:
QDataStream& operator <<(QDataStream& out, QVector<double> vec);
But this is not working as well. I know I can write data to my file from
QVector
by going through it one by one, but my data vectors are way too long (in millions) and so I am looking for a faster way to write it to a binary file. Is there any way I can write an entireQVector
to a binary file without iterating through it? -
@J-Hilk I am unable to find any examples to tell me how to get the header. The
QDataStream
class description only says just a little about serializingQVectors
do you know any examples that I can follow so that I have an idea how to read my binary file? -
@CJha said in How to write entire QVector to a binary file?:
so that I have an idea how to read my binary file?
Read it with QDataStream. The format is internal.
-
@Christian-Ehrlicher Hi, I cannot, I have to read it in Matlab.
-
@CJha said in How to write entire QVector to a binary file?:
I have to read it in Matlab.
Then create your own format or check what matlab needs to read it with it's built-in functions.
-
@J-Hilk But this is internal and may change (although the chance is very very low)
-
@J-Hilk Thanks, I will go through the document and see if it can be helpful to me.
@Christian-Ehrlicher @JonB The only other way is to iterate through aQVector<double>
, and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly. -
@CJha said in How to write entire QVector to a binary file?:
and is a time-consuming process, especially with millions of data points, and that's why I am interested in doing it directly.
Did you look at the QDataStream implementation? It does exactly the same... so why should this be faster?
-
@CJha said in How to write entire QVector to a binary file?:
I have to read it in Matlab.
If I remeber correctly Matlab reads csv (aka text) so
QTextStream
is what you want. something like:QFile file("BinTest.bin"); file.open(QIODevice::WriteOnly | QIODevice::Text); QTextStream out(&file); QVector<double> vec; for(int ii = 0; ii < 10; ++ii){ vec << ii * 0.33; } for(auto&& val : vec) out << val << ','; file.close();
-
@Christian-Ehrlicher I didn't know that before I saw the source code for
QDataStream
. I assumed since for aQVector
data points are in adjacent memory positions pushing an entire vector into a binary file must be faster than iterating over it. -
@CJha said in How to write entire QVector to a binary file?:
The only other way is to iterate through a QVector<double>, and is a time-consuming process, especially with millions of data points
Yep. That's what has to be done, and it's what the
<<
serializer does, as @J-Hilk showed you. The only other way would be if you can get the address of contiguousQVector
memory and save from there, which I'm guessing can be done. However......If @VRonin's latest post is correct and you're supposed to produce text instead to export then you cannot help but do it one-by-one....
P.S.
BTW, you'd have to test, but my guess is that code to output data points one-by-one instead of in a contiguous clump is not what would be slow over 1,000,000 points --- rather, the size of the output written to file will be what is significant.... -
You're correct with the adjacent memory but QVector can also hold objects so memcpy'ing it out will not work there. You can use memcpy if you want in your case but QDataStream is generic and has no optimizations for such things.
-
@VRonin Yes, but writing a .csv file takes much longer than writing a binary file (almost 10 times more for large data sets). I have tried and tested it. I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours, and this file will be analysed in Matlab by researchers. If I write 1 million data points in a .csv file it takes around 4 seconds while doing the same in a binary file takes around 400 milliseconds.
-
@CJha said in How to write entire QVector to a binary file?:
I am gathering data at a much faster rate, up to 1 million doubles per second and I have to write it to a file continuously for hours
actually stop right here!
if this program is used more than once, you're going to destroy your HD/SSD very quickly!
I'm sure there's an other - in memory - way to hand over those data points