File Copy Question



  • Hi I am writing a file replicator program that replicates the contents of a source directory to a destination directory.

    The copy progress is not consistent when copying from a system SSD to a backup mechanical drive. It will copy a good 100 MB or so then pause a fifth of a second or so then start up again. I get the same results disabling AV and firewall. When reading from the SSD only without doing any file writing it's consistent. When going from backup drive to system SSD it is consistent as well.

    On another Vista system when copying from the slow computer to a flash drive it pauses every couple seconds I suspect to clear some blocks for writing but other than that it behaves normally. When copying from network to a Vista computer drive the copy is consistent.

    I suspect the first inconsistency is due to the lack of buffering during file copying. My current copy code is below. buffer is a QByteArray and targetTempFile is a QFile. The count integer is just leftover from a previous implementation.

    How come QT doesn't provide progress tracking with the bytesWritten() signal for QFile copying? It looks like this code just bouncing back and forth between reading the source and writing the target right? Do you think a read/write buffer would maximize performance and provide more consistent copy peformance? I found this guy - it appears this function is buffered correct? If so and there are no complications I will give it a go. Anything else to know? Thanks!

            for (int count = 0; !(buffer = sourceFile.read(1000000)).isEmpty() && cancel == false; count++)
            {
                int readSize = buffer.size();
                targetTempFile.write(buffer);
                if (advanced = true)
                    setAdvancedProgressMeters(readSize, sourceFile.size(), job.getSize());
                setAdvancedTimeLabels(startTime.elapsed());
            }




  • Thanks but how do I track copy progress with that function?


  • Lifetime Qt Champion

    Hi,

    There's no monitoring you can do with that one.

    In any case, expecting that the behaviour should be the same accros all types of media and network etc. is wrong. The drives themselves have buffers, you also have to into account the bus used to communicate with your different disk, the drivers, the system load, the OS type, the speed of the hard drive motors, etc. All these factors and others can alter the speed of copy.

    [edit: fixed missing no...]



  • How do I monitor/track progress that one?


  • Lifetime Qt Champion

    Sorry, there was a missing no in that sentence.



  • Do you think a buffered IO operation might perform better though instead of bouncing back and forth between reading/writing data? I think like a producer/consumer type of thing.


  • Lifetime Qt Champion

    It's basically the same expect that you split the operation between two threads. Though it's worth to try if only to benchmark.



  • How come QFile copy doesn't emit the writesbytten signal? Also with a producer consumer buffer there'd be concurrent read/writes don't you think it'd raise performance significantly perhaps almost by a factor of 2?


  • Lifetime Qt Champion

    Because that wouldn't make sense. bitesWritten is for when you write in a device. With a copy QFile writes in another device.

    Again: benchmark. Parallelising disk write doesn't necessarily means that you'll double anything. If you have only one disk on the receiving side, writing twice as much data in parallel won't have the performance boost you'd expect.


  • Qt Champions 2016

    Hi
    Just as a note:
    If i was to write anything for file copy on the windows platform
    i would look in this tool source
    https://ipmsg.org/tools/fastcopy.html.en

    I use it to copy array of mirrors ( 8 TB ) to different backup locations and
    fastcopy delivers flawless and in high speed. :)



  • Thanks mrjj definitely worth checking out.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.