Writing to file in chunks
-
Currently I'm writing a huge
QByteArray
to a file after filling it out with data:QByteArray my_ba; // Fill out my_ba in some loops path = "/some/path.data"; QFile file(path); file.open(QIODevice::WriteOnly); file.write(my_ba); file.close();
Since my
QByteArray
can be in order of GBs, to reduce memory usage, I need to write myQByteArray
to file in chunks.Does Qt have any convenient tool to do so? Is there any standard practice?
-
hi @m3g1dd
well QByteArray hasremove
that shortes the QByteArray and returns the removed bytes as a new QByteArray
so something like this could work:while (!myByteArray.isEmpty()) { file.write(myByteArray.remove(0, chunksize)); file.flush(); //needed or QFile will wait untill the eventloop executes, befor actually writing }
-
@m3g1dd said in Writing to file in chunks:
Since my QByteArray can be in order of GBs, to reduce memory usage, I need to write my QByteArray to file in chunks.
I don't get what you say here. You already have your
QByteArray
in memory, so you have already used up "GBs of memory". Why do you think writing that to a file will use any further memory (e.g. do you think it will take a copy?), and so why would "writing it in chunks" reduce memory? (I actually think it would increase memory as getting a chunk from aQByteArray
has to allocate more memory.)P.S.
No offence to @J-Hilk, but I'd be very careful if you're going to use hismyByteArray.remove(0, chunksize)
, or anything else which manipulates theQByteArray
content. Unless you understand the implementation of the precise call you make, you will be in imminent danger of causing Qt to create a copy/move memory etc. on yourQByteArray
, and at GBs size that could be expensive... (e.g.QByteArray::remove()
does alter theQByteArray
, at very best I think that's a largememmove()
, at worst it's amemcpy()
). -
I ended up writing to file in chunks periodically like this:
for(unsigned int j = 0; j < Count; ++j) { // Fill out `QByteArray my_ba` in each iteration of the loop // But write my_ba only periodically and then clear it! // period is picked in such a way that writing to file will be done for every 1MB of data if( j % period == 0){ if(file){ file->write(my_ba); file->flush(); my_ba.clear(); } } // ... }
I clear my
QByteArray
periodically by doingmy_ba.clear()
after writing its content to file, therefore myQByteArray
will never get big and its memory consumption is hence reduced.