Writing to file in chunks

[m3g1dd]

Currently I'm writing a huge QByteArray to a file after filling it out with data:

QByteArray my_ba;

// Fill out my_ba in some loops

path = "/some/path.data";
QFile file(path);
file.open(QIODevice::WriteOnly);
file.write(my_ba);
file.close();

Since my QByteArray can be in order of GBs, to reduce memory usage, I need to write my QByteArray to file in chunks.

Does Qt have any convenient tool to do so? Is there any standard practice?

[J.Hilk]

hi @m3g1dd
well QByteArray has remove that shortes the QByteArray and returns the removed bytes as a new QByteArray
so something like this could work:

while (!myByteArray.isEmpty()) {
     file.write(myByteArray.remove(0, chunksize));
     file.flush(); //needed or QFile will wait untill the eventloop executes, befor actually writing
}

[JonB]

@m3g1dd said in Writing to file in chunks:

Since my QByteArray can be in order of GBs, to reduce memory usage, I need to write my QByteArray to file in chunks.

I don't get what you say here. You already have your QByteArray in memory, so you have already used up "GBs of memory". Why do you think writing that to a file will use any further memory (e.g. do you think it will take a copy?), and so why would "writing it in chunks" reduce memory? (I actually think it would increase memory as getting a chunk from a QByteArray has to allocate more memory.)

P.S.
No offence to @J-Hilk, but I'd be very careful if you're going to use his myByteArray.remove(0, chunksize), or anything else which manipulates the QByteArray content. Unless you understand the implementation of the precise call you make, you will be in imminent danger of causing Qt to create a copy/move memory etc. on your QByteArray, and at GBs size that could be expensive... (e.g. QByteArray::remove() does alter the QByteArray, at very best I think that's a large memmove(), at worst it's a memcpy()).

[mrjj]

Hi
Adding to @JonB , im not following why writing in chunks should lower
the overall memory consumption.
Why not write data data to file in the loops that fills out
the my_ba ?
Unless you need my_ba for something else. :)

[m3g1dd]

I ended up writing to file in chunks periodically like this:

for(unsigned int j = 0; j < Count; ++j) {

// Fill out `QByteArray my_ba` in each iteration of the loop

    // But write my_ba only periodically and then clear it!
    // period is picked in such a way that writing to file will be done for every 1MB of data
    if( j % period == 0){
            if(file){
                file->write(my_ba);
                file->flush();
                my_ba.clear();
            }
        }
 
    // ...
}

I clear my QByteArray periodically by doing my_ba.clear() after writing its content to file, therefore my QByteArray will never get big and its memory consumption is hence reduced.

[m3g1dd]

@J.Hilk said in Writing to file in chunks:

file.flush();

Thanks for the file.flush() hint, I didn't know that :)

[JonB]

@m3g1dd
file.flush() will not do anything about memory consumption. You only need it (though it does no harm) if you intend to look at the file's content from another program while your application is running.