File Copy Buffer Question

[Crag_Hack]

Hi I have a quick question - I am writing a file replicator data backup software and am experiencing a 'flaw' in the data copying. If I copy data from an SSD to a HDD the SSD is much faster to read data than the HDD is to write data. Because of this the data that is read fills a buffer before it is written to the HDD. This wouldn't be a problem except that the progress meter for data copy stalls when the buffer gets filled then when it gets emptied it races to get filled again. So the progress meter indicates a bunch of progress, then it stalls, then it indicates progress, then stalls, etc. It looks pretty ugly compared to constant progress. Is there a way around this with Qt? I tried using QFile.flush() but it didn't help.
Thanks!

[Crag_Hack]

I timed all operations using my smartphone timer. CopyFileEx with buffer was the smoothest performing for the progress bars. Here are the rough results in seconds:
for ~1 GB data split among 14 files:
windows copy times: 23 20 21
using QByteArrays as in third post in this thread: 21 21 22
using CopyFileEx no buffer 21 24 23
using CopyFileEx with buffer 21.21 20.64 21.37
for 512 files each 12 KB in size:
using QByteArrays as in third post in this thread: 7.21 7.68 7.28 6.78
windows copy times: 5.31 4.76 4.75 5.41
using CopyFileEx no buffer: 11.40 10.60 10.17
using CopyFileEx with buffer: 9.18 8.96 10.19

[SGaist]

Hi,

How are you implementing the copy ?

[Crag_Hack]

Sorry I forgot the code :) sourceFile and targetTempFile are QFile.

QByteArray buffer;
for (int count = 0; !(buffer = sourceFile.read(1000000)).isEmpty() && cancel == false; count++)
{
    int readSize = buffer.size();
    targetTempFile.write(buffer);
    //targetTempFile.flush();
    setProgressMeters(readSize, fileSize, jobSize);
    setTimeLabels(startTime.elapsed(),readSize);
}

[SGaist]

sourceFile is a QFile ?

[Crag_Hack]

Yes that's correct.

[SGaist]

What you could do is base your progress bar on the percentage of data copied rather than the size of the file or the number of blocks to copy.

[Crag_Hack]

Thanks but that would still stall when the buffer fills right?

[SGaist]

There's not much you can do if the system slows down, but that way you avoid the "chaotic" update.

[Crag_Hack]

I think the best way is to use Win API CopyFileEx with no buffer flag set.

[JKSH]

@Crag_Hack said in File Copy Buffer Question:

So the progress meter indicates a bunch of progress, then it stalls, then it indicates progress, then stalls, etc. It looks pretty ugly compared to constant progress.

Look at the bright side: Windows users are used to this already so they won't complain too much ;)

(source: https://www.xkcd.com/612/)

@Crag_Hack said in File Copy Buffer Question:

I think the best way is to use Win API CopyFileEx with no buffer flag set.

Check the copy speed with and without the buffer. Make sure there's no significant difference in speed.

[Crag_Hack]

I timed all operations using my smartphone timer. CopyFileEx with buffer was the smoothest performing for the progress bars. Here are the rough results in seconds:
for ~1 GB data split among 14 files:
windows copy times: 23 20 21
using QByteArrays as in third post in this thread: 21 21 22
using CopyFileEx no buffer 21 24 23
using CopyFileEx with buffer 21.21 20.64 21.37
for 512 files each 12 KB in size:
using QByteArrays as in third post in this thread: 7.21 7.68 7.28 6.78
windows copy times: 5.31 4.76 4.75 5.41
using CopyFileEx no buffer: 11.40 10.60 10.17
using CopyFileEx with buffer: 9.18 8.96 10.19

[Crag_Hack]

I forgot to mention, thanks for the comedy ! :)