Best way to compress data to append to a file?

[Crag_Hack]

I have a quick question - what's the best Qt way to append data to a file and compress that data? Right now I just do:
QFile file;
QDataStream stream(file);
stream << data;
Thanks!

[Bonnie]

I think this need several steps.

1. Create a QDataStream that operates on a QByteArray(or QBuffer), write your data using that QDataStream to the QByteArray.
2. use qCompress to compress the QByteArray above and you will get a new compressed QByteArray.
3. Open your file with write permission + append flag and create a new QDataStream to operate on it as you already did, write the compressed QByteArray using QDataStream .

So when you read your file, you have to follow the reversed steps:

1. Use QDataStream to read your file to QByteArray(s)
2. For each QByteArray, you need to use qUncompress to get a uncompressed QByteArray.
3. For each uncompressed QByteArray, use QDataStream to read data from it.

qCompress and qUncompress use zlib and are well-wrapped for QByteArray. You can also use other libraries if you want but you need to wrap the API by yourself.

[JKSH]

@Crag_Hack said in Best way to compress data to append to a file?:

what's the best Qt way to append data to a file

You need to tell us what kind of file it is, or describe the file format.

...and compress that data?

Compress the data before writing to the file? Or compress the file itself?

See https://doc.qt.io/qt-5/qbytearray.html#qCompress as a starting point.

[Crag_Hack]

Currently without compression the file is a QFile linked with a QDataStream. To process the data I open the file with open(QIODevice::WriteOnly | QIODevice::Append). Then I pass the data to the stream linked to the file with stream << data where data is a QVector of a custom data type I use in my program. I need to compress the file to decrease the data size since it can become quite bloated. Ideally I would like to append the compressed data to the file instead of writing to a file then compressing it and storing the results in another file. Is this possible? Is there another way perhaps more effective?
Thanks again

[Bonnie]

I think this need several steps.

1. Create a QDataStream that operates on a QByteArray(or QBuffer), write your data using that QDataStream to the QByteArray.
2. use qCompress to compress the QByteArray above and you will get a new compressed QByteArray.
3. Open your file with write permission + append flag and create a new QDataStream to operate on it as you already did, write the compressed QByteArray using QDataStream .

So when you read your file, you have to follow the reversed steps:

1. Use QDataStream to read your file to QByteArray(s)
2. For each QByteArray, you need to use qUncompress to get a uncompressed QByteArray.
3. For each uncompressed QByteArray, use QDataStream to read data from it.

qCompress and qUncompress use zlib and are well-wrapped for QByteArray. You can also use other libraries if you want but you need to wrap the API by yourself.

[JKSH]

To be clear, "append" usually means the file already contains existing data and you want to add more data to it. @Crag_Hack:

• Do you want to write new data to a file that already contains data? OR
• Do you want to create a new file altogether? OR
• Do you want to erase the old data in an existing file and replace it with new data?

[Crag_Hack]

@Bonnie Thanks will try it out
@JKSH The first, I want to add data to a file that already exists and contains data unless of course this is the first write to that file in which case just create it with the new data.

[JKSH]

@Crag_Hack said in Best way to compress data to append to a file?:

I want to add data to a file that already exists and contains data unless of course this is the first write to that file in which case just create it with the new data.

OK.

Be aware that each time you append new data, it is added as a separate chunk. So, when you read the data from the file again, you must call stream >> bytearray multiple times (1 time per chunk)

[Pablo J. Rogina]

@Crag_Hack two things please.
If this issue is solved, please mark this post as such.
Then create another issue for the new question.

[JKSH]

@Pablo-J-Rogina said in Best way to compress data to append to a file?:

Then create another issue for the new question.

Thanks, @Pablo-J-Rogina. The new question is moved to https://forum.qt.io/topic/117460/how-to-skip-chunks-using-qdatastream