File Copy Question

Crag_Hack

Hi I am writing a file replicator program that replicates the contents of a source directory to a destination directory.

The copy progress is not consistent when copying from a system SSD to a backup mechanical drive. It will copy a good 100 MB or so then pause a fifth of a second or so then start up again. I get the same results disabling AV and firewall. When reading from the SSD only without doing any file writing it's consistent. When going from backup drive to system SSD it is consistent as well.

On another Vista system when copying from the slow computer to a flash drive it pauses every couple seconds I suspect to clear some blocks for writing but other than that it behaves normally. When copying from network to a Vista computer drive the copy is consistent.

I suspect the first inconsistency is due to the lack of buffering during file copying. My current copy code is below. buffer is a QByteArray and targetTempFile is a QFile. The count integer is just leftover from a previous implementation.

How come QT doesn't provide progress tracking with the bytesWritten() signal for QFile copying? It looks like this code just bouncing back and forth between reading the source and writing the target right? Do you think a read/write buffer would maximize performance and provide more consistent copy peformance? I found this guy - it appears this function is buffered correct? If so and there are no complications I will give it a go. Anything else to know? Thanks!

        for (int count = 0; !(buffer = sourceFile.read(1000000)).isEmpty() && cancel == false; count++)
        {
            int readSize = buffer.size();
            targetTempFile.write(buffer);
            if (advanced = true)
                setAdvancedProgressMeters(readSize, sourceFile.size(), job.getSize());
            setAdvancedTimeLabels(startTime.elapsed());
        }

? Offline

Hi! You can just use bool QFile::copy(const QString &fileName, const QString &newName) [static].

Crag_Hack

Thanks but how do I track copy progress with that function?

SGaist

Hi,

There's no monitoring you can do with that one.

In any case, expecting that the behaviour should be the same accros all types of media and network etc. is wrong. The drives themselves have buffers, you also have to into account the bus used to communicate with your different disk, the drivers, the system load, the OS type, the speed of the hard drive motors, etc. All these factors and others can alter the speed of copy.

[edit: fixed missing no...]

Crag_Hack

How do I monitor/track progress that one?

SGaist

Sorry, there was a missing no in that sentence.

Crag_Hack

Do you think a buffered IO operation might perform better though instead of bouncing back and forth between reading/writing data? I think like a producer/consumer type of thing.

SGaist

It's basically the same expect that you split the operation between two threads. Though it's worth to try if only to benchmark.

Crag_Hack

How come QFile copy doesn't emit the writesbytten signal? Also with a producer consumer buffer there'd be concurrent read/writes don't you think it'd raise performance significantly perhaps almost by a factor of 2?

SGaist

Because that wouldn't make sense. bitesWritten is for when you write in a device. With a copy QFile writes in another device.

Again: benchmark. Parallelising disk write doesn't necessarily means that you'll double anything. If you have only one disk on the receiving side, writing twice as much data in parallel won't have the performance boost you'd expect.

mrjj

Hi
Just as a note:
If i was to write anything for file copy on the windows platform
i would look in this tool source
https://ipmsg.org/tools/fastcopy.html.en

I use it to copy array of mirrors ( 8 TB ) to different backup locations and
fastcopy delivers flawless and in high speed. :)

Crag_Hack

Thanks mrjj definitely worth checking out.