Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

Launching a queue of processes (was: "Working with Qthreads")



  • Hello Everyone!

    I want to create an application which will create(dynamically) n number of threads for a set of tasks .
    When one thread finishes its task it will get on to next queued up task.

    Something like this:

      thread1(task1)
      thread2(task2)
      thread3(task3)
    

    if thread1 finishes first then it with proceed with the next task

      thread1(task2)
    

    or another thread will be created for that purpose.

    Thank you!


  • Lifetime Qt Champion

    Hi,

    Sounds like you should take a look at QtConcurrent.



  • From what I have been reading, I think I need QThreadPool, but I can't seem to find any example that I can understand how it works . This stackoverflow answer is the closest I got so far to be able to understand it.

    I want exactly this behavior:

    600px-Thread_pool.svg.png


  • Moderators

    @hbatalha can you provide more details about your tasks?

    For example, if you want to run the same function on every element in a vector, you would use QtConcurrent::map(): https://doc.qt.io/qt-5/qtconcurrentmap.html QtConcurrent takes care of the QThreadPool behind the scenes, so you don't need to manage the pool yourself.



  • [ @hbatalha can you provide more details about your tasks? ]
    --Every new thread will have a process that it will run.

    I managed to do it (correct me if it is not the best way)

    class Task : public QRunnable
    {
        QString id;
    public:
        Task(QString i) : id(i) {}
    
    protected:
        void run() override
        {
            QString exe = "some_process.exe";
    
            QStringList args;
    
            args << "some_args" << i;
    
            QProcess *process = new QProcess;
    
            process->setProcessChannelMode(QProcess::MergedChannels);
    
            process->start(exe, args);
        }
    };
    
    // I will call it here
    
    void Main::startProcesses()
    {
            QThreadPool *pool = new QThreadPool(this);
         
             pool->setMaxThreadCount(3);
         
             for(auto const& e: selections/*QStringList*/)
             {
                 Task *task = new Task(e)
                 pool->start(task);     
             }        
    }
    
    

    That is what I got so far.



  • @hbatalha said in Working with Qthreads:

    That is what I got so far.

    Why do you use threads to start a process?
    This is not really useful and just add complexity to your application in my eyes.



  • @hbatalha
    One of my bug-bears is that it seems these days that just about every beginner to Qt starts out by trying to use threads. Which are hard to get right, and nearly always turn out to be unnecessary anwyay.

    Please answer @KroMignon's question first, before going any further. If you are wanting to create multiple QProcess to run OS commands, why do you want threads at all?



  • @KroMignon said in Working with Qthreads:

    Why do you use threads to start a process?

    Now that I think about it, it doesn't really seem logical (cause it it isn't?).
    What I want is that I only have 3 processes running at one point in time, the first thing that came to mind was threads, and I can see that this is the wrong way to do it.

    From what I am seeing is that I every time I call pool->start(task) the process will begin and I will end up with all of the processes running at the same time instead of only 3 (please correct me if I am wrong).

    So, how can I achieve what I want, having only 3 processes running at one point in time?


  • Lifetime Qt Champion

    @hbatalha said in Working with Qthreads:

    having only 3 processes running at one point in time?

    Simply start 3 QProcess instances without any threads.



  • @hbatalha
    As @Christian-Ehrlicher has said. The point is, with Qt when you go QProcess::start(...) this starts the process, but already does not block. The sub-process runs asynchronously, and emits signals when e.g. it has finished (do not use the blocking QProcess::waitForFinished() method). This means that your 3 sub-processes already all run at the same time, without you needing any threads in Qt.



  • @Christian-Ehrlicher said in Working with Qthreads:

    Simply start 3 QProcess instances without any threads.

    How can I make that when one process finishes, another onewill be started?



  • @hbatalha
    Like I said, use the signal QProcess::finished. When one sends that, start a new one. That of course makes your processes execute sequentially, instead of at the same time.



  • @JonB said in Working with Qthreads:

    @hbatalha
    Like I said, use the signal QProcess::finished. When one sends that, start a new one. That of course makes your processes execute sequentially, instead of at the same time.

    Can you provide some example code? I have a basic idea one what you are saying but not enough to translate it to code.



  • @hbatalha said in Working with Qthreads:

    Can you provide some example code? I have a basic idea one what you are saying but not enough to translate it to code.

    Qt framework is an asynchronous framework, this helps for example for UI to avoid locking the main thread and enable user interactions.

    One way to do it would be to create a list parameters for the process you want to start and go through this list to start each process at the end of previous one.

    For example:

    struct ProcInfo
    {
        QString procName;
        QStringList procParameters;
        // add what ever you need
    };
    

    Then use this to start each process after the other:

    class Launcher : public QObject
    {
        Q_OBJECT
    
       QProcess mProc;
       QList<ProcInfo> mProcToStart; 
    public:
       Launcher(QObject * parent = nullptr) : QObject(parent)
       {
           mProc.setProcessChannelMode(QProcess::MergedChannels);
           connect(&mProc, &QProcess:stateChanged, [this](QProcess::ProcessState newState) {
               if(newState == QProcess::NotRunning)
                   startNext();
               });
       }
    
       void startProcs(const QList<ProcInfo> &toStart)
       {
           mProcToStart = toStart;
           startNext();
       }
       void startNext()
       {
            if(mProcToStart .isEmpty())
                return;
            auto nextP = mProcToStart.takeFirst();
            mProc.start(nextP.procName, nextP.procParameters);
       }
    };
    

    Something like that, up to you to finish it ;)


  • Moderators

    @hbatalha said in Working with Qthreads:

    @JonB said in Working with Qthreads:

    Like I said, use the signal QProcess::finished. When one sends that, start a new one. That of course makes your processes execute sequentially, instead of at the same time.

    Can you provide some example code? I have a basic idea one what you are saying but not enough to translate it to code.

    First, get familiarized with the concept of signals and slots: https://doc.qt.io/qt-5/signalsandslots.html This is a core part of most Qt applications. When you know how to use signals and slots, many things will become clearer.



  • @KroMignon said in Working with Qthreads:

    Then use this to start each process after the other:

    Is it possible to run multiple processes at the same time.

    Also, I couldn't get the code to work, maybe I am missing something or I am using it wrong:

        QList<ProcInfo> procs;
        Launcher l;
        for(auto const& e: selections)
        {
            QStringList args;
    
            args << "-some_args" << e;
    
            procs.push_back({"program.exe", args});
        }
    
        l.startProcs(procs);
    

  • Lifetime Qt Champion

    @hbatalha said in Working with Qthreads:

    Is it possible to run multiple processes at the same time.

    As I already said - simply create more than one QProcess instance.



  • @JKSH said in Working with Qthreads:

    First, get familiarized with the concept of signals and slots: https://doc.qt.io/qt-5/signalsandslots.html This is a core part of most Qt applications. When you know how to use signals and slots, many things will become clearer.

    I did it and was able to do this, which works(is it correct?)

    void Dialog::on_pushButton_3_clicked()
    {
        for(int i = 0; i < 3; ++i)
        {
            QStringList args;
    
            args << "some_args" << selections.takeFirst();
    
            QProcess* pro = new QProcess;
    
            pro->setProcessChannelMode(QProcess::MergedChannels);
    
            connect(pro, &QProcess::stateChanged, [this](QProcess::ProcessState newState)
            {
                if(newState == QProcess::NotRunning)
                    start_next_process();
            });
    
            pro->start("program.exe", args);
        }
    }
    
    void Dialog::start_next_process()
    {
        if(!selections.isEmpty())
        {
            QStringList args;
    
            args << "some_args" << selections.takeFirst();
    
            QProcess *process = new QProcess;
    
            connect(process, &QProcess::stateChanged, [this](QProcess::ProcessState newState)
            {
                if(newState == QProcess::NotRunning)
                    start_next_process();
            });
    
            process->setProcessChannelMode(QProcess::MergedChannels);
    
            process->start("prgram.exe", args);
        }
    }
    

    I took at look at this but haven't yet completely understood



  • @hbatalha
    You are working from @KroMignon's template. You will not need the QProcess::finished signal I mentioned/your link now refers to because he is effectively doing this (indirectly) via the QProcess::stateChanged signal he connects instead. Which is fine.

    Have a look again at his code. In yours you have effectively copied the same block of code out of start_next_process() and into your on_pushButton_3_clicked(). You do not need to do that (repeating the same code is always a suspicious sign). His code is designed for you to use a selections member variable to "queue up" the 3 (in your case) processes you'd like it to run. Your code does no mesh well with that. Please take the time to understand how @KroMignon's approach works.



  • This post is deleted!


  • A process is a program that is running on your computer. This can be anything from a small background task, such as a spell-checker or system events handler to a full-blown application like Internet Explorer or Microsoft Word. All processes are composed of one or more threads.

    I was under the impression that OP want to
    run an individual SINGLE thread in sequence in SINGLE process.

    So where do all these "queued multiple instance processes " come from ?
    Would it be possible to get back to the original question ?



  • @JonB said in Working with Qthreads:

    Please take the time to understand how @KroMignon's approach works.

    I have read and I partially understand it. What I can't seem to be able to figure out is how does it work well with my code. when I call it via the void startProcs(const QList<ProcInfo> &toStart) method nothing happens, I have tried using it like this:

        QList<ProcInfo> mProcToStart;
    
        for(int i = 0; i < 3; ++i)
        {
            QStringList args;
    
            args << "some_args" << selections.takeFirst();
    
            mProcToStart.push_back({"programs.exe", args});
        }
    
        Launcher launch;
    
        launch.startProcs(mProcToStart);
    

    What am missing here?

    When it does work, will it run only the three queued processes taken from selections or all of the selections members?



  • @hbatalha said in Working with Qthreads:

    when I call it via the void startProcs(const QList<ProcInfo> &toStart) method nothing happens

    What do you expect to happen, which doesn't happen?

    Step through it in a debugger, or put qDebug() statements at judicious places. That's what programming is all about!



  • @hbatalha said in Working with Qthreads:

    What am missing here?
    When it does work, will it run only the three queued processes taken from selections or all of the selections members?

    I think your knowledge about how Qt works is to "light".
    Perhaps you should take time to read some parts of Qt documentation:

    The example class I have written it very lightweight and is just an implementation example to do what you want to do (at least what I have understood):

    • have a list of process to start
    • launch next process when first is finished

    How does it work:
    the class holds a QProcess member which will start each process and a list which defines the process to be started.
    In constructor, the signal QProcess::stateChanged is used to be informed when the current process is finished and then start next in queue.

    Before starting the next process, you could, for example, check the results of previous process, with:

    • mProc.exitCode() to get application exit code
    • mProc.readAll() to get application output

    You could also add signals to Launcher class to be informed about all process done or to give progression, and what ever you need.

    Again, this is just a code skeleton, to give you a starting point.



  • @JonB said in Working with Qthreads:

    What do you expect to happen, which doesn't happen?

    I expect tha program.exe will start but it doesn't.

    Step through it in a debugger, or put qDebug() statements at judicious places. That's what programming is all about!

    I did , nothing seems to happen when it gets to mProc.start(nextP.procName, nextP.procParameters);. It just finishes up the three elements in the list



  • @KroMignon said in Working with Qthreads:

    I think your knowledge about how Qt works is to "light".

    I agree, I will do just that.

    The example class I have written it very lightweight and is just an implementation example to do what you want to do (at least what I have understood):

    What I want to achieve is, have a QStringList (selections) with unique arguments id, each argument will be used to start a process in its due time.

    However, I want theses processes running concurrently but only a given number at one point in time, when one process finishes another one will be started.

    For example, suppose selections has 5 elements {"1", "2", "3", "4", "5"}, the element 1, 2, 3 will be the first to start the processes and these processes will run at the same time and when e.g. the process with the argument 1 finishes another process, this time with the argument 4, will be started.

    This code does exactly that, and as pointed out by @JonB it has repeated code and he suggested me trying to understand your example and work with it. That is what I am trying to do but as beginner to GUI programming and QT it is proving to be quite challenging.

    mProc.exitCode() to get application exit code
    mProc.readAll() to get application output

    Tried both, exitcode was 0(zero), and the readAll() shows nothing.

    Can you provide an example on how you intended the code to be used?



  • @hbatalha said in Working with Qthreads:

    I expect tha program.exe will start but it doesn't.

    Your code does not try to execute program.exe, it tries to execute programs.exe or prgram.exe. You don't have any code for checking for errors/output, you wouldn't be informed. Even if it did work, unless you say how you know it has not be run it is not evident you would be sure it had not run. A proper version should use the various other signals of QProcess; and/or put a qDebug() statement into the lambda in your connect() so that you see all QProcess:stateChanged transitions.

    Glancing now at @KroMignon's code, I'm not sure how it's supposed to work as-is :) The Launcher() constructor does the connect() for the single process mProc, while startNext() re-uses that mProc to launch each process. I'm not sure you're supposed to re-use an existing QProcess instance while it has a process running. But he did say "Something like that, up to you to finish it ;)".

    His code may work only if you run one process at a time, and wait for one to finish before starting the next. Your start_next_process(), which you say does work, creates a new QProcess for each process, so may be more successful in this case.

    In principle to get 3 going you should be able to put (at least) 3 into the queue and then call start_next_process() 3 times. Thereafter as one finishes a new one will be pulled and started. I agree you would have to look at how his code works/change it to allow for this. But even with your code earlier, because the body of on_pushButton_3_clicked() is so similar to start_next_process() it looks to me as though on_pushButton_3_clicked() could simply call start_next_process() 3 times rather than repeating code.


  • Moderators

    This post is deleted!


  • @hbatalha said in Working with Qthreads:

    However, I want theses processes running concurrently but only a given number at one point in time, when one process finishes another one will be started.

    As I wrote before, this was only a basic skeleton, and as @JonB supposed, you can not reuse a QProcess to start another process. That is why it runs only once.

    So I change the class to create a new QProcess on each process start:

    class Launcher : public QObject
    {
        Q_OBJECT
    
       QList<ProcInfo> mProcToStart;
       QProcess *mProc;
    public:
       explicit Launcher(QObject * parent = nullptr) : QObject(parent), mProc(nullptr)
       {
       }
       ~Launcher()
       {
           if(mProc)
               mProc->deleteLater();
       }
    
       void startProcs(const QList<ProcInfo> &toStart)
       {
           mProcToStart.append(toStart);
           startNext();
       }
       bool startNext()
       {
           if(mProcToStart.isEmpty() || (mProc && mProc->state() != QProcess::NotRunning))
               return false;
           auto nextP = mProcToStart.takeFirst();
           mProc = new QProcess();
           mProc->setProcessChannelMode(QProcess::MergedChannels);
           connect(mProc, &QProcess::stateChanged, [this](QProcess::ProcessState newState) {
               if(newState == QProcess::NotRunning)
               {
                   qDebug() << "Process" << mProc->program() << "with arguments" << mProc->arguments()<< "done";
                   qDebug() << "Exit code is:" << mProc->exitCode();
                   qDebug() << "Returned data:" << qUtf8Printable(QString::fromLocal8Bit(mProc->readAll()));
                   qDebug() << "--------------------------------------";
    
                   mProc->deleteLater();
                   mProc = nullptr;
                   if(!startNext())
                       emit isDone();
               }
           });
           mProc->start(nextP.procName, nextP.procParameters);
           return true;
       }
    
    signals:
       void isDone();
    };
    

    And here is a working example of use:

    int main(int argc, char *argv[])
    {
        QCoreApplication a(argc, argv);
    
        QList<ProcInfo> procs;
        QStringList defArgs;
        defArgs
                << "-n" << "1"  // Only once
                << "-w" << "5000"; // wait up to 5 seconds
        procs.push_back({"ping.exe", QStringList() << defArgs <<"google.com" });
        procs.push_back({"ping.exe", QStringList() << defArgs <<"amazon.com" });
        procs.push_back({"ping.exe", QStringList() << defArgs <<"forum.qt.io" });
    
        Launcher l;
        l.startProcs(procs);
        QObject::connect(&l, &Launcher::isDone, &a, &QCoreApplication::quit);
    
        return a.exec();
    }
    

    But once again, this is only a "playground". Up to you to do adapt this or create a new class to fit your needs.



  • @JonB said in Working with Qthreads:

    Your code does not try to execute program.exe

    It does, I mistyped.

    you don't have any code for checking for errors/output, you wouldn't be informed.

    I created the code to check the output, which showed nothing and checked the exitcode too.

    unless you say how you know it has not be run it is not evident you would be sure it had not run

    The program when it is finished it will have created files in a specific directory. So I go into that directory to check if it has run.

    I'm not sure you're supposed to re-use an existing QProcess instance while it has a process running.

    It has to finish first, right? Maybe that is what is stopping it to work correctly. As I said, I am a complete beginner to Qt, I am still picking up things as I go and you guys are being of great help.

    it looks to me as though on_pushButton_3_clicked() could simply call start_next_process() 3 times rather than repeating code.

    I did just that, works perfect, thank you.



  • @KroMignon said in Working with Qthreads:

    So I change the class to create a new QProcess on each process start:

    create a new QProcess was one of the many things I tried to see if I could get it to work but now I see I did that wrong.

    But somehow it is not working with my code, now it is crashing the entire entire application and I can't seem to figure out why. I tried running it as a console application but it is complaining that I isDone is undefined.

    So, I won't be using your code for now until I gather more knowledge of Qt and also as I have a working code that seems to be ok. But I do want to use it as a console application so I can understand it better and learn from it.

    As a final request before marking this thread as solved. I would like to know when am I to use this kind of approach or similar.

    This is my working code now:

    void Dialog::on_pushButton_3_clicked()
    {
        for(int i = 0; i < 3; ++i)
                start_next_process();
    }
    
    void Dialog::start_next_process()
    {
        if(!selections.isEmpty())
        {
            QStringList args;
    
            args << "some_args" << selections.takeFirst();
    
            QProcess *process = new QProcess;
    
            connect(process, &QProcess::stateChanged, [this](QProcess::ProcessState newState)
            {
                if(newState == QProcess::NotRunning)
                    start_next_process();
            });
    
            process->setProcessChannelMode(QProcess::MergedChannels);
    
            process->start("program.exe", args);
        }
    }
    

    What are the differences between your approach and this? ( @JonB I would appreciate hearing this from you also) .

    Edit: I got the console application working, put the class in its .h file and now it works.



  • @hbatalha
    This looks fine. You should free the QProcess you create via new QProcess. Perhaps your lambda could add process->deleteLater() when NotRunning.



  • @JonB said in Working with Qthreads:

    @hbatalha
    This looks fine. You should free the QProcess you create via new QProcess. Perhaps your lambda could add process->deleteLater() when NotRunning.

    I thought that every object created via new in Qt would be handled somehow by the Qt application itself so the user won't have to worry about memory management (this is what I got from a YouTube video, maybe I got it wrong).

    So basically every time, I create a a process via new I will have to add this code?

            connect(process, &QProcess::stateChanged, [this](QProcess::ProcessState newState)
            {
                if(newState == QProcess::NotRunning)
                {
                    process->deleteLater();
                }
            });
    

    or something like that?



  • @hbatalha said in Working with Qthreads:

    or something like that?

    You are close, you need to capture also process in the lambda:

    connect(process, &QProcess::stateChanged, [this, process](QProcess::ProcessState newState)
            {
                if(newState == QProcess::NotRunning)
                {
                    process->deleteLater();
                }
            });
    


  • @KroMignon
    The compiler is complaining about unused this.

    dialog.cpp:51:48: warning: lambda capture 'this' is not used
    

    Is it ok if I remove it when I don't have any this's method call inside the lambda?



  • @hbatalha
    In theory "warning" is just that and ignoring it "should not" affect the outcome of the code.
    Your mileage will vary...

    I am still not comfortable with lambda , but if you are looking for whatever ( process, thread etc) to conclude why not say so?

    such as
    if process done
    clean-up whatever

    it seems redundant to

    if process changed
    if process done
    cleanup later

    Just saying...
    Best of luck

    PS
    I still think you be better off using QConcurrent - definitely less messy...
    Your dime...


  • Lifetime Qt Champion

    @hbatalha said in Launching a queue of processes (was: "Working with Qthreads"):

    Is it ok if I remove it when I don't have any this's method call inside the lambda?

    Yes, if you do not access any non-static members/methods.



  • @hbatalha said in Launching a queue of processes (was: "Working with Qthreads"):

    I thought that every object created via new in Qt would be handled somehow by the Qt application itself so the user won't have to worry about memory management (this is what I got from a YouTube video, maybe I got it wrong).

    Yes and no :)

    Because QProcess derives from QObject, it has a constructor QProcess::QProcess(QObject *parent = nullptr). If you specified, say, this as parent (new QProcess(this)) that would make your Dialog be its parent. This would be an improvement, since when Dialog is destroyed it will take each of the created QProcesss with it, preventing a memory leak. (I would probably do this here anyway.) And in many cases in Qt that is enough, e.g. for widget children on a widget parent.

    Here, however, at least theoretically your code allows hundreds of QProcresss to be created with the Dialog as parent if it stays there a long time. That would mean that the memory/resources used by each QProcess would persist as long as the dialog is open. Yet we actually finish with each QProcess once it has run the program and finished. Hence we should free them at that point, rather than later on. And for that we use QObject::deleteLater(), for safety.



  • @JonB
    Your explanation is crystal clear. Thank you, for all the help.



  • @jsulm said in Launching a queue of processes (was: "Working with Qthreads"):

    Yes, if you do not access any non-static members/methods.

    thanks

    @AnneRanch said in Launching a queue of processes (was: "Working with Qthreads"):

    but if you are looking for whatever ( process, thread etc) to conclude why not say so?

    I didn't understand what you meant.

    I still think you be better off using QConcurrent - definitely less messy...

    Can you elaborate on why you think so.

    I read a little about QConcurrent and it doesn't seem to be able to do what I want to achieve.
    If you disagree can you please provide some code?!


Log in to reply