Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

QFile, QThread, QConcurrent and how to Qt



  • Hello everyone,

    Hope you are all safe and healthy in these trying times.

    I have just joined the community and I am thrilled to be here.
    For the past month I went through several video courses(over a hundred of hours) related to C++ and QT C++, a lot of QT docs, forum posts, books and so on as I really want to board the Qt ship. I am eager to move on with the shiny UI stuff but I really want to strengthen my fundaments first. That said, I have no previous coding experience besides the trivial exercises after some of the video lectures so please bear with me.
    I am trying to do baby steps here as my impression for coding UIs so far is mostly dealing with threads and QIODevice, thus I wanted to start off with something more simple like a file IO tool and build my way up to a network tool, and then move on with life and the fancy UI stuff.
    If you trust there is a better way of training the basic/fundamental things I am certainly opened for suggestions :)

    In the last few days I've been working on the said file io test tool and needless to say I have tons of questions. Although I can make it work for the most part some things are still unclear to me, nor am I certain of any best practices.

    Without further ado I'd like to make some queries based on your invaluable experience.

    main just "manager m;" and loops indefinitely awaiting sting input which fires one of the two slots in the manager class.

    #ifndef MANAGER_H
    #define MANAGER_H
    
    #include <QObject>
    #include <QDebug>
    #include <QtConcurrent>
    #include <QThread>
    #include <QSharedDataPointer>
    #include "worker.h"
    
    class manager : public QObject
    {
        Q_OBJECT
    public:
        explicit manager(QObject *parent = nullptr);
        ~manager();
    
    signals:
        void start();
        void stop();
    
    public slots:
        void process();
        void quit();
        void test();
    
    private:
        void createWorker(manager* newWorker);
        QThread* timerThread = new QThread(this);
        QTimer * timer = new QTimer(0);
    
    };
    
    #endif // MANAGER_H
    
    #include "manager.h"
    
    manager::manager(QObject *parent) : QObject(parent)
    {
        qInfo() << "Creating manager class on: " << QThread::currentThread();
        timer->setInterval(1000);
        timer->moveToThread(timerThread);
        QObject::connect(timerThread, SIGNAL(started()), timer, SLOT(start()),Qt::QueuedConnection);
        QObject::connect(timerThread, SIGNAL(finished()), timer, SLOT(stop()),Qt::QueuedConnection);
        timerThread->start();
    
    }
    
    manager::~manager()
    {
        quit();
    }
    
    void manager::process()
    {
        QtConcurrent::run(this, &manager::createWorker, this);
        emit start();
    }
    
    void manager::quit()
    {
        qInfo() << "Stopping the manager";
        stopit_now = true;
        emit stop();
        timerThread->quit();
    
    }
    
    void manager::test()
    {
        qInfo() << "test from manager";
    }
    
    void manager::createWorker(manager *newManager)
    {
        qInfo() << "Creating worker";
    
        worker* newWorker = new worker(0,8,8192,1); //variables for testing purposes
    
        connect(newManager,&manager::start, newWorker, &worker::start, Qt::QueuedConnection);
        connect(newManager,&manager::stop, newWorker, &worker::stop, Qt::QueuedConnection);
    
        connect(timer, &QTimer::timeout, newWorker, &worker::currSpeed, Qt::DirectConnection);
    
        newWorker->start();
    
    }
    
    
    #ifndef WORKER_H
    #define WORKER_H
    
    #include <atomic>
    #include <QFile>
    #include <QDir>
    #include <QIODevice>
    #include <QDateTime>
    #include <QByteArray>
    
    #include <QObject>
    #include <QDebug>
    #include <QTimer>
    #include <QElapsedTimer>
    #include <QThread>
    #include <QThreadPool>
    #include <QFuture>
    #include <QFutureWatcher>
    #include <QtConcurrent>
    
    #include <QEventLoop>
    //used these while building the whole thing for testing purposes and changed them with input from the constructor
    static qint64 bufferSize = 0x800000;
    static QByteArray test(bufferSize, '0');
    
    extern bool stopit_now;
    
    class worker : public QObject
    {
        Q_OBJECT
    public:
        explicit worker(QObject *parent = nullptr, quint64 IO_SIZE = 0, quint64 FILE_SIZE = 0, int WORKER_NUMBER = 1);
        ~worker();
    
    //global variables that each worker can update
        static std::atomic<quint64> current_speed;
        static std::atomic<quint64> total_bytes_written;
    
    signals:
    
    public slots:
        void start();
        void stop();
        void currSpeed();
    
    private:
        QEventLoop loop;
        quint64 ioSize;
        quint64 fileSize;
        QByteArray writeBuffer;
        int workerNumber;
    
    
        bool writeData_singleFile(QString path, QByteArray data, QIODevice::OpenMode);
    
    };
    
    #endif // WORKER_H
    
    
    #include "worker.h"
    std::atomic<quint64> worker::current_speed;
    std::atomic<quint64> worker::total_bytes_written;
    bool stopit_now = false;
    worker::worker(QObject *parent, quint64 IO_SIZE, quint64 FILE_SIZE, int WORKER_NUMBER)
        : QObject(parent), ioSize(IO_SIZE*1024*1024), fileSize(FILE_SIZE*1024*1024), workerNumber(WORKER_NUMBER)
        // I will make some changes here so it can work with bytes and kbytes
    {
        QObject::setObjectName("Worker #" + QString::number(workerNumber));
        this->writeBuffer.fill('0', ioSize);
        qInfo() << this << " constructed on: " << QThread::currentThread();
    }
    
    worker::~worker()
    {
        qInfo() << this << " deconstructed on: " << QThread::currentThread();
    }
    
    
    void worker::start()
    {
        qInfo() << this << " started on: " << QThread::currentThread();
    
        // doing heavy lifting
        QString path = QDir::currentPath() + QDir::separator() + "test.tst";
        QFutureWatcher<void> watcher;
        QThreadPool pool;
        QFuture<void> future = QtConcurrent::run(this, &worker::writeData_singleFile,path,writeBuffer,QIODevice::WriteOnly);
        qInfo() << "Back in worker -> start thread";
        watcher.setFuture(future);
        watcher.waitForFinished();
    
        qInfo() << this << " Worker finished on: " << QThread::currentThread();
        this->stop();
    }
    
    void worker::stop()
    {
        qInfo() << this << " stopping on: " << QThread::currentThread();
        stopit_now = true;
        delete this;
    }
    
    void worker::currSpeed()
    {
        qInfo() << "Worker " << this << " current speed: " << current_speed/1024/1024 << "MB/s";
        current_speed = 0;
    }
    
    bool worker::writeData_singleFile (QString path, QByteArray data, QIODevice::OpenMode mode) {
        qInfo() << "writeData_singleFile function running on: " << QThread::currentThread();
    
        QFile file(path);
        qInfo() << "Creating file: " << path;
        if(file.open(mode))
        {
            qInfo() << "File opened";
            qInfo() << "IO Size specified: " << data.size();
            qInfo() << "File size specified: " << fileSize;
        } else {
            qWarning() << "File cannot be opened: " << file.errorString();
            return false;
        }
    
    
        QElapsedTimer timer;
        timer.start();
    
        for (size_t i = 0; i < fileSize/data.size(); i++)
        {
            if(stopit_now) { qInfo() << "STOP call issued - Stopping writes!"; break; }
    // the check below appears to be slowing down everything by 50%, i.e. instead of getting 100MB/s results for current_speed I get 50MB/s.
    // I suppose because the check is done just as many as the writes it cripples them but how I would know if any write has failed?
    //        if(!file.write(data)) {
    //            qInfo() << "Failed to write data to file: " << file.errorString();
    //            file.close();
    //        }
            current_speed += file.write(data);
            //file.flush();
        }
    
        // perhaps there is a better way to do this, without filling up an entire array?
        // the idea of this function is to be able complete the file with the designated size, instead of creating weird-sized files
        // if there is a better way to handle this scenario please do tell :)
        if (fileSize % data.size()) {
            data.fill(fileSize % data.size());
            if(!file.write(data)) {
                qInfo() << "Failed to write data to file: " << file.errorString();
                file.close();
            }
            current_speed += file.write(data);
        }
    
        qInfo() << "File created successfully";
        total_bytes_written = this->ioSize + this->fileSize;
        qInfo() << total_bytes_written << " written to file";
    
        file.flush();
        file.close();
    
        qInfo() << "Worker Time elapsed: " << timer.elapsed()/1000;
        if(total_bytes_written != 0) qInfo() << "Average Speed: " << (total_bytes_written/1024/1024)/(timer.elapsed()/1000) << "MB/s";
    
        // Eventually I will do the same function for the reads or reuse the same with an if statement to switch from write to read depending on the QIODevice open mode.
    
        return true;
    }
    
    
    

    From what I have seen thus far if my file is less than several GBs(possibly depending on the machine RAM/cache) it gets "written" in less than a second as I believe it gets cached due to some sort of internal Windows/NTFS mechanisms.
    As I am currently working and testing on Windows according to the QT docs QIODevice::Unbuffered not supported on Windows. Just thinking out loud here but I suppose this explains my issue above. If so, is there any way around that, as I ideally want to be able to switch between buffered and unbuffered IO regardless of the OS platform?
    I have tried calling file.flush() after each write but it doesn't change the behavior and my total metrics, such as average speed(calculating total bytes writen divided by elapsed time) are having unreal numbers with smaller file.

    Also, I have seen some performance tools having an option of using concurrent IO from a single worker(instead of multiple workers) to fully saturate a high performance pipe. I would imagine QTConcurrent would facilitate that but I am not sure how. Do I have to use QDataStream to achieve something similar, or is there another way to have QFile write to the same file from multiple threads? Eventually, if this is indeed possible I might not even use multiple files to simulate reading/writing multiple files.
    In either case I would appreciate some insight here.

    I have finally made the QTimer works as I want it(I think) but I am not sure if it's the correct way of doing things. To get it going I had to:

        QThread* timerThread = new QThread(this);
        QTimer * timer = new QTimer(0);
    

    but I can't wrap my head around just yet as to why I had to do that since by my manager class and QThread are inheriting QObject. The same applies for QTimer. Why do I have to give my class object as parent for the new thread and null the timer?

    As soon as my worker starts it gets right into the heavy lifting and the timer timeout signal would never call the worker slot, which is why I have to use the Qt::DirectConnection to have the slot executed on the timer thread, as the timer thread is free and not blocked by operations like the worker thread. Is this an accurate assumption on my part?

    Furthermore, for signal/slot connections made in my manager class constructor(not sure if it's the best idea) in this particular case I had to do:

        QObject::connect(timerThread, SIGNAL(started()), timer, SLOT(start()),Qt::QueuedConnection);
        QObject::connect(timerThread, SIGNAL(finished()), timer, SLOT(stop()),Qt::QueuedConnection);
    

    which took some time to figure out(not the problem here) but then again I am not sure why the usual approach like the one below is not working:

       connect(newManager,&manager::start, newWorker, &worker::start, Qt::QueuedConnection);
       connect(newManager,&manager::stop, newWorker, &worker::stop, Qt::QueuedConnection);
    

    Could you please elaborate on that?

    Most of the time multiple workers(or a single worker doing concurrent writes if I manage to solve that part) will be used, so with the my current knowledge and skill the one thing I thought of using is static class variables so they can be shared by all workers. As this is not thread safe some articles imply that I should use atomic variables, as volatile might not be safe with modern hardware. Mutexes seem to be an option reading the QT docs but as I understand it is more taxing as opposed to the std::atomic approach. At my level I am not really sure which approach would be better and why.

    Additionally, once I get to the GUI part I would want to extend the program to be able to manipulate individual workers on demand, i.e. to be able to stop any worker at any time. If by any chance I am not missing something with QTConcurrent am I on a right track assuming that the way to go is creating new threads manually and then moving the individual workers to the new threads, thus calling thread.quit() to stop the given worker if need be?

    As this is my first real program I am sure it is full of holes, leaks, and overall room for improvement so any guidance and pointers based on the code written and in general would be much appreciated.

    I wouldn't mind if you could also suggest some good learning/training materials related to Qt, regardless if they are paid(so long they are worth it) or not.
    As of now I am stocked with several QML/Widget courses from Udemy, Qt5 Cadaques, Advanced Qt Programming Creating Great Software with C++ and Qt 4, C++ GUI Programming with Qt 4, Second Edition and Learn Qt5. Although I am settled for a while with the above if you guys have any good recommendations don't hold them back :)

    Thanks in advance for your patience and enlightenment, best regards and stay safe.


  • Lifetime Qt Champion

    Hi and welcome to devnet,

    You have a good plan which is great.

    My first recommendation: do not start learning C++ together with threading. It's like trying to start learning to pilot a plane using a loaded fighter jet in a combat zone.

    For example, you are using several different threading techs together while it is not needed at all. QtConcurrent already provides asynchronous support that you are short circuiting in your secondary thread.

    You do not need to go that low level before making GUIs. There all sorts of tools that you can build. Take something you are interested in and then check what tech you would need and go from there.

    You can also check with projects like KDE that are great for newcomers to get started because there are lots of different tools, applications and libraries that can help you learn how to code while helping the project as well :-)



  • @SGaist said in QFile, QThread, QConcurrent and how to Qt:

    My first recommendation: do not start learning C++ together with threading. It's like trying to start learning to pilot a plane using a loaded fighter jet in a combat zone.

    :D



  • Hello and thank you for the input.

    A plane is a plane - you die unless you land properly or eject. Whether you are in the combat zone or not you either learn flying or die trying. The combat zone will surely make you a better pilot if you live that long tho =)

    I do understand I am using different technologies, but as I am still learning I want to test how they all work - together or separately, hence the reason I asked if in the long run would be better to manually create threads and move workers on them manually so I can be able to manage them, instead of using QtConcurrent.

    My drive is not to master the GUI but rather have a good enough fundament to be able to manage it properly, and more importantly connect it with an adequate Qt C++ backend. In other words I am not exactly interested in making shiny, flashy, glamorous GUIs that do not do anything and someone else has to connect the dots. What I want is to be able to be proficient in the core and be able to create and manage a decent UI, as I am still trying to find my place in this world.

    I am still interested in some enlightenment related to all topics I asked though, in case someone willing to walk me through.

    Stay safe


  • Lifetime Qt Champion

    @A-Newbie said in QFile, QThread, QConcurrent and how to Qt:

    I do understand I am using different technologies, but as I am still learning I want to test how they all work - together or separately, hence the reason I asked if in the long run would be better to manually create threads and move workers on them manually so I can be able to manage them, instead of using QtConcurrent.

    Here is the answer you won't like: it all depends on the thing you want to do.

    In any case, you would usually not invoke QtConcurrent code from a separate thread since QtConcurrent provides a high level set of functions that does the threading for you. To illustrate the issue, you are calling run and right after that you call waitForFinished on the future watcher. This defeats the purpose of using QtConcurrent since you are blocking the execution until run has ended.

    You should by learning the way Qt implement its asynchronous paradigm using signals and slots. This will allow you to have a better grasp on how to use QtConcurrent properly.

    You can then play with QThread, separately from QtConcurrent. You'll see that you do not need to use them both at the same time the way you did.



  • Hi, and thanks again.

    I don't mind the answer at all. On the contrary I am aware that everything depends on what you really want to do so apologies if my explanations or questions are/were vague/incorrect. This is the reason why I tried to explain in more details the purpose of the code I posted and how I would want to possibly evolve it but I guess I didn't do a very good job :<

    I know the thread is blocked, as explained in the initial post and I want my thread to be blocked(or I think I do) until it is finished as I want the worker as an object to self-destroy once it's done doing it's job, or to be able to interrupt the worker's job at any point if need be. If I only do the QtConcurrent run on the write function it will execute in a separate thread(than the worker's thread) and I(personally) don't know how to control that thread if it has to be interrupted or something of the sort. This is the reason why I asked if there is a way to control the QtConcurrent thread(s), or in the long run if I want control my threads to use QThread and move the worker to the newly created thread, thus be able to stop the worker/thread at any time using QThread's signals/slots.
    If there is a more elegant way to handle this scenario by all means please do tell.

    I just want to know how much I can do with either. For example, I would go any day with QtConcurrent if I simply want to create 1000 files and fill them up with raw data, but then again I am not sure how achieve the same concurrency/async logic if I simply want to fill or read from/to a single file. I mean yes, I can fire the same read/write function in QtConcurrent but I can hardly imagine the OS letting me write chunks of data from multiple threads as they will eventually overlap and cause file corruption(with a normal file), which means if the OS wouldn't let me do that the only other thing that makes sense to me is QDataStream instead of QFile for the task, but since this is simply a speculation on my behalf I really wanted someone to shed more light in this regard.

    I am still interested in those two as well:

    I have finally made the QTimer works as I want it(I think) but I am not sure if it's the correct way of doing things. To get it going I had to:

    QThread* timerThread = new QThread(this);
    QTimer * timer = new QTimer(0);
    

    but I can't wrap my head around just yet as to why I had to do that since by my manager class and QThread are inheriting QObject. The same applies for QTimer. Why do I have to give my class object as parent for the new thread and null the timer?

    Furthermore, for signal/slot connections made in my manager class constructor(not sure if it's the best idea) in this particular case I had to do:

    QObject::connect(timerThread, SIGNAL(started()), timer, SLOT(start()),Qt::QueuedConnection);
    QObject::connect(timerThread, SIGNAL(finished()), timer, SLOT(stop()),Qt::QueuedConnection);
    

    which took some time to figure out(not the problem here) but then again I am not sure why the usual approach like the one below is not working:

    connect(newManager,&manager::start, newWorker, &worker::start, Qt::QueuedConnection);
    connect(newManager,&manager::stop, newWorker, &worker::stop, Qt::QueuedConnection);
    Could you please elaborate on that?

    I do understand these things might be trivial for most people here but wrapping my head around these for me is like jumping few levels at once so the extra nudge here would be highly appreciated.


  • Lifetime Qt Champion

    @A-Newbie said in QFile, QThread, QConcurrent and how to Qt:

    I mean yes, I can fire the same read/write function in QtConcurrent but I can hardly imagine the OS letting me write chunks of data from multiple threads

    Why do you want to write to same file from different threads? Only one thread should do the writing. Other threads can communicate with that thread via signals/slots to provide the data to write.

    "Why do I have to give my class object as parent for the new thread and null the timer?" - you don't have to, you can. Check https://doc.qt.io/qt-5/objecttrees.html Short explanation: if parent ("this" in QThread(this)) is deleted it deletes its children (QThread instance in this case). If you do the memory management by yourself you do not have to pass parent.

    "null the timer?" - why do you think you have to pass null? You can, but then you have to delete it manually later. Or you pass "this", then the timer will be deleted as soon as "this" is deleted.


  • Moderators

    @A-Newbie
    If I may suggest a git repo of mine:
    https://github.com/DeiVadder/QtThreadExample

    In that project I do a threading task in each of the "Qt"-ways
    The related functions are numbered.

    May help you further in your quest for knowledge 😉



  • Hello again,

    @jsulm
    I am not sure if I want to use many threads(among other things) to write to the same file, which is the reason I am asking how performance tools achieve this exactly. Many data IO performance bench tools have an option called "overlapped IO and IO depth", "concurrent IO count", "Outstanding IO" or something of the sort. In either case increasing the IO numbers to something adequate, say 3 or 4 brings about better read/write performance results from a single file, while a single-threaded sequential IO cannot do that.
    I can probably simulate the same performance results if I shoot multiple threads reading from or writing to multiple files, but I am struggling at this point to understand how to do it with a single file.

    As for the new thread and timer, while I am aware of the parent-child relationship, but for some reason this timer wasn't working before I did that. Might have been just a coincidence in case I had my Qt::Connection was set to Queued instead of Direct at the time, or messing something some signals/slots up, or something of the sort, but I did spend the whole night reading forums as my timer wasn't working if moved to a different thread.
    The reason I didn't pass my class as a parent to my timer is because of the Thread Affinity explained here: https://doc.qt.io/qt-5/qobject.html assuming I got the explanation there right of course =)
    I can swear I didn't touch my signals/slots and tried Queued vs Direct connection every time I was changing something else.
    In either case by all means it seems I goofed up something so I will settle with that, as it explains why I couldn't explain what happened there ^_^

    @J-Hilk
    Thanks a bunch, having some good examples really means a lot to me! It gave me some good insight, but I mean to ask about the subclassing from QThread method. I read a lot of posts implying this is not the best method to handle threads. Is this because you have to be more wary of where and how you create things when you inherit from QThread in general and people struggle with this part, or is it something else?

    Also there is something which @SGaist said and has been bugging me:
    "In any case, you would usually not invoke QtConcurrent code from a separate thread since QtConcurrent provides a high level set of functions that does the threading for you."
    Besides the obvious reason, namely taking one additional thread(more resources) for the task what are the downfalls here? Is it something in general or more of a Qt thread methods/technology related?

    Any clues about why I had to use two different signal/slot connection methods?


  • Lifetime Qt Champion

    @A-Newbie said in QFile, QThread, QConcurrent and how to Qt:

    Any clues about why I had to use two different signal/slot connection methods?

    What exactly did not work with the other approach?


  • Moderators

    @A-Newbie said in QFile, QThread, QConcurrent and how to Qt:

    mean to ask about the subclassing from QThread method. I read a lot of posts implying this is not the best method to handle threads. Is this because you have to be more wary of where and how you create things when you inherit from QThread in general and people struggle with this part, or is it something else?

    Well in general it is very easy to make something wrong. You have to have an fundamental understanding of Qt Object, &- Parent/Child system Threadaffinity in combination to the caller etc.

    If you do an endless loop you'll have to implement your own check & abort functionality and if you call exec() you might as well do the Worker approach.

    Besides the obvious reason, ..., what are the downfalls here

    QtConcurrent provides, amongst other things, an integrated ThreadPool management, which is nice, but the whole thing comes with an overhead, so much in fact, that QtConcurrent is its own module, whereas QThread is part of QtCore

    Any clues about why I had to use two different signal/slot connection methods?

    care to elaborate?



  • @J-Hilk
    Thank you for the insight kind sir, much appreciated.

    @jsulm and @J-Hilk

    Looking at my manager.cpp file

    30090985-07d7-4859-a3ae-2b75d926adf1-image.png

    What am I doing wrong here?
    I know I have missed something in the docs, but I can't seem to find out what yet.


  • Moderators

    @A-Newbie start is overloaded, it excepts an int and a void parameter
    The docs actually have an example for exactly this one,
    https://doc.qt.io/qt-5/qtimer.html#details

    on how to correctly call the overload in connect

    QTimer *timer = new QTimer(this);
        connect(timer, &QTimer::timeout, this, QOverload<>::of(&AnalogClock::update));
        timer->start(1000);
    


  • @J-Hilk
    Thanks again, I got it going with QOverload<>::of , but I am confused. Since I am passing it a void signal shouldn't it automatically choose the void version of the slot as normal overloads do?
    Do the macro version, which I actually used do some sort of additional checks and adjustments and it works fine, or was it just luck it worked out for me and it chose the correct slot version?


  • Lifetime Qt Champion

    @A-Newbie
    Hi
    The difference between new syntax (the one needed overload )
    and SIGNAL/SLOT macros is how that the new syntax works compile-time and the old
    macro-based , works runtime.
    The new syntax is based on pointers and needs full type info, the
    old Macro version looks up the names at runtime.

    more info:
    https://wiki.qt.io/New_Signal_Slot_Syntax

    You can see why the new syntax needs the QOverload in cases where there is
    signals with the same name but different parameters. like int and string.



  • @mrjj
    Hi and thanks, this was an eye opener as I had no idea about the whole overloading thing. Never met it in any of the learning materials I went through so far.

    Regardless, I got answers pretty much to all of my questions and everyone helped a lot. Is there a way to simply have this topic as solved without picking a specific answer?


  • Moderators

    @A-Newbie said in QFile, QThread, QConcurrent and how to Qt:

    Regardless, I got answers pretty much to all of my questions and everyone helped a lot.

    great 👍

    Is there a way to simply have this topic as solved without picking a specific answer?

    sure bottem right side, topic tools button, simply set it to solved there


Log in to reply