Data Block Copy Question

[Crag_Hack]

I have a quick question. I have a data backup program that reads source files in QByteArray blocks of at most 1,000,000 bytes in size. Might I be able to irk out more performance by using a smaller or larger block size? Does it depend on the source and target media and/or the file size?
Thanks!

[SGaist]

Hi,

You should do benchmarks.

Speed depends on:

• device type (Hard Drive, SSD, Flash, NAND, MMC, etc.)
• bus type (USB, FireWire, ATA, SATA, etc.)
• storage technology (RAID0-5, JBOD, etc.)
• File system type (ext2,3,4, ReiserFS, ZFS, exFAT, APFS, HFS+, etc)
• System load
• Network load if using a NAS

[VRonin]

if you are taking a copy of a file from one place to the other, using QFile::copy instead might turn out to be faster

[Crag_Hack]

Thanks SGaist. Any numbers you suggest starting with? Will large amounts of small files have different properties than larger files?

[Crag_Hack]

Thanks VRonin. I need to track progress so have to use block by block.

[SGaist]

Yes they will but it will also depend on your file system. Having lots of small files means that you will have more IOCTL calls to open and close the files so more "times lost" there.

[VRonin]

@Crag_Hack said in Data Block Copy Question:

I need to track progress so have to use block by block.

UNTESTED

#include <QCoreApplication>
#include <QTimer>
#include <QFile>
#include <QFileInfo>
#include <QFuture>
#include <QDebug>
#include <QFutureWatcher>
#include <QtConcurrent>
int main(int argc, char *argv[])
{
    QCoreApplication a(argc,argv);
    QTimer progressTimer;
    const QString source("mySourceFile.dat");
    const QString destination("myDestFile.dat");
    const qint64 sourceSize = QFileInfo(source).size();
    QEventLoop pauser;
    QFutureWatcher<void> signaler;
    QObject::connect(&signaler,&QFutureWatcher<void>::finished,&pauser,&QEventLoop::quit);
    QFileInfo destInfo(destination);
    QObject::connect(&progressTimer,&QTimer::timeout,[&destInfo,&sourceSize]()->void{
        destInfo.refresh();
        qDebug() << "Progress: " << static_cast<qint64>(100)*destInfo.size()/sourceSize << "%";
    });
    const QFuture<void> future = QtConcurrent::run(std::bind(&QFile::copy,source,destination));
    signaler.setFuture(future);
    progressTimer.start(50);
    pauser.exec();
    progressTimer.stop();
    return 0;
}

[kshegunov]

Interesting approach. One could probably also monitor the file changes through QFileSystemWatcher instead of a timer.

[Crag_Hack]

Wow VRonin you got creative :)