Data Block Copy Question

Crag_Hack

I have a quick question. I have a data backup program that reads source files in QByteArray blocks of at most 1,000,000 bytes in size. Might I be able to irk out more performance by using a smaller or larger block size? Does it depend on the source and target media and/or the file size?
Thanks!

SGaist

Hi,

You should do benchmarks.

Speed depends on:

device type (Hard Drive, SSD, Flash, NAND, MMC, etc.)
bus type (USB, FireWire, ATA, SATA, etc.)
storage technology (RAID0-5, JBOD, etc.)
File system type (ext2,3,4, ReiserFS, ZFS, exFAT, APFS, HFS+, etc)
System load
Network load if using a NAS

VRonin

if you are taking a copy of a file from one place to the other, using QFile::copy instead might turn out to be faster

Crag_Hack

Thanks SGaist. Any numbers you suggest starting with? Will large amounts of small files have different properties than larger files?

Crag_Hack

Thanks VRonin. I need to track progress so have to use block by block.

SGaist

Yes they will but it will also depend on your file system. Having lots of small files means that you will have more IOCTL calls to open and close the files so more "times lost" there.

VRonin

@Crag_Hack said in Data Block Copy Question:

I need to track progress so have to use block by block.

UNTESTED

#include <QCoreApplication>
#include <QTimer>
#include <QFile>
#include <QFileInfo>
#include <QFuture>
#include <QDebug>
#include <QFutureWatcher>
#include <QtConcurrent>
int main(int argc, char *argv[])
{
    QCoreApplication a(argc,argv);
    QTimer progressTimer;
    const QString source("mySourceFile.dat");
    const QString destination("myDestFile.dat");
    const qint64 sourceSize = QFileInfo(source).size();
    QEventLoop pauser;
    QFutureWatcher<void> signaler;
    QObject::connect(&signaler,&QFutureWatcher<void>::finished,&pauser,&QEventLoop::quit);
    QFileInfo destInfo(destination);
    QObject::connect(&progressTimer,&QTimer::timeout,[&destInfo,&sourceSize]()->void{
        destInfo.refresh();
        qDebug() << "Progress: " << static_cast<qint64>(100)*destInfo.size()/sourceSize << "%";
    });
    const QFuture<void> future = QtConcurrent::run(std::bind(&QFile::copy,source,destination));
    signaler.setFuture(future);
    progressTimer.start(50);
    pauser.exec();
    progressTimer.stop();
    return 0;
}

kshegunov

Interesting approach. One could probably also monitor the file changes through QFileSystemWatcher instead of a timer.

Crag_Hack

Wow VRonin you got creative :)