Solved Writing to file in chunks
-
Currently I'm writing a huge
QByteArray
to a file after filling it out with data:QByteArray my_ba; // Fill out my_ba in some loops path = "/some/path.data"; QFile file(path); file.open(QIODevice::WriteOnly); file.write(my_ba); file.close();
Since my
QByteArray
can be in order of GBs, to reduce memory usage, I need to write myQByteArray
to file in chunks.Does Qt have any convenient tool to do so? Is there any standard practice?
-
hi @m3g1dd
well QByteArray hasremove
that shortes the QByteArray and returns the removed bytes as a new QByteArray
so something like this could work:while (!myByteArray.isEmpty()) { file.write(myByteArray.remove(0, chunksize)); file.flush(); //needed or QFile will wait untill the eventloop executes, befor actually writing }
-
@m3g1dd said in Writing to file in chunks:
Since my QByteArray can be in order of GBs, to reduce memory usage, I need to write my QByteArray to file in chunks.
I don't get what you say here. You already have your
QByteArray
in memory, so you have already used up "GBs of memory". Why do you think writing that to a file will use any further memory (e.g. do you think it will take a copy?), and so why would "writing it in chunks" reduce memory? (I actually think it would increase memory as getting a chunk from aQByteArray
has to allocate more memory.)P.S.
No offence to @J-Hilk, but I'd be very careful if you're going to use hismyByteArray.remove(0, chunksize)
, or anything else which manipulates theQByteArray
content. Unless you understand the implementation of the precise call you make, you will be in imminent danger of causing Qt to create a copy/move memory etc. on yourQByteArray
, and at GBs size that could be expensive... (e.g.QByteArray::remove()
does alter theQByteArray
, at very best I think that's a largememmove()
, at worst it's amemcpy()
). -
Hi
Adding to @JonB , im not following why writing in chunks should lower
the overall memory consumption.
Why not write data data to file in the loops that fills out
the my_ba ?
Unless you need my_ba for something else. :) -
I ended up writing to file in chunks periodically like this:
for(unsigned int j = 0; j < Count; ++j) { // Fill out `QByteArray my_ba` in each iteration of the loop // But write my_ba only periodically and then clear it! // period is picked in such a way that writing to file will be done for every 1MB of data if( j % period == 0){ if(file){ file->write(my_ba); file->flush(); my_ba.clear(); } } // ... }
I clear my
QByteArray
periodically by doingmy_ba.clear()
after writing its content to file, therefore myQByteArray
will never get big and its memory consumption is hence reduced. -
@J.Hilk said in Writing to file in chunks:
file.flush();
Thanks for the
file.flush()
hint, I didn't know that :) -
@m3g1dd
file.flush()
will not do anything about memory consumption. You only need it (though it does no harm) if you intend to look at the file's content from another program while your application is running.