Launching a queue of processes (was: "Working with Qthreads")

hbatalha

Hello Everyone!

I want to create an application which will create(dynamically) n number of threads for a set of tasks .
When one thread finishes its task it will get on to next queued up task.

Something like this:

  thread1(task1)
  thread2(task2)
  thread3(task3)

if thread1 finishes first then it with proceed with the next task

  thread1(task2)

or another thread will be created for that purpose.

Thank you!

SGaist

Hi,

Sounds like you should take a look at QtConcurrent.

hbatalha

From what I have been reading, I think I need QThreadPool, but I can't seem to find any example that I can understand how it works . This stackoverflow answer is the closest I got so far to be able to understand it.

I want exactly this behavior:

600px-Thread_pool.svg.png

JKSH

@hbatalha can you provide more details about your tasks?

For example, if you want to run the same function on every element in a vector, you would use QtConcurrent::map(): https://doc.qt.io/qt-5/qtconcurrentmap.html QtConcurrent takes care of the QThreadPool behind the scenes, so you don't need to manage the pool yourself.

hbatalha

[ @hbatalha can you provide more details about your tasks? ]
--Every new thread will have a process that it will run.

I managed to do it (correct me if it is not the best way)

class Task : public QRunnable
{
    QString id;
public:
    Task(QString i) : id(i) {}

protected:
    void run() override
    {
        QString exe = "some_process.exe";

        QStringList args;

        args << "some_args" << i;

        QProcess *process = new QProcess;

        process->setProcessChannelMode(QProcess::MergedChannels);

        process->start(exe, args);
    }
};

// I will call it here

void Main::startProcesses()
{
        QThreadPool *pool = new QThreadPool(this);
     
         pool->setMaxThreadCount(3);
     
         for(auto const& e: selections/*QStringList*/)
         {
             Task *task = new Task(e)
             pool->start(task);     
         }        
}

That is what I got so far.

KroMignon

@hbatalha said in Working with Qthreads:

That is what I got so far.

Why do you use threads to start a process?
This is not really useful and just add complexity to your application in my eyes.

JonB

@hbatalha
One of my bug-bears is that it seems these days that just about every beginner to Qt starts out by trying to use threads. Which are hard to get right, and nearly always turn out to be unnecessary anwyay.

Please answer @KroMignon's question first, before going any further. If you are wanting to create multiple QProcess to run OS commands, why do you want threads at all?

hbatalha

@KroMignon said in Working with Qthreads:

Why do you use threads to start a process?

Now that I think about it, it doesn't really seem logical (cause it it isn't?).
What I want is that I only have 3 processes running at one point in time, the first thing that came to mind was threads, and I can see that this is the wrong way to do it.

From what I am seeing is that I every time I call pool->start(task) the process will begin and I will end up with all of the processes running at the same time instead of only 3 (please correct me if I am wrong).

So, how can I achieve what I want, having only 3 processes running at one point in time?

Christian Ehrlicher

@hbatalha said in Working with Qthreads:

having only 3 processes running at one point in time?

Simply start 3 QProcess instances without any threads.

JonB

@hbatalha
As @Christian-Ehrlicher has said. The point is, with Qt when you go QProcess::start(...) this starts the process, but already does not block. The sub-process runs asynchronously, and emits signals when e.g. it has finished (do not use the blocking QProcess::waitForFinished() method). This means that your 3 sub-processes already all run at the same time, without you needing any threads in Qt.

hbatalha

@Christian-Ehrlicher said in Working with Qthreads:

Simply start 3 QProcess instances without any threads.

How can I make that when one process finishes, another onewill be started?

JonB

@hbatalha
Like I said, use the signal QProcess::finished. When one sends that, start a new one. That of course makes your processes execute sequentially, instead of at the same time.

hbatalha

@JonB said in Working with Qthreads:

@hbatalha
Like I said, use the signal QProcess::finished. When one sends that, start a new one. That of course makes your processes execute sequentially, instead of at the same time.

Can you provide some example code? I have a basic idea one what you are saying but not enough to translate it to code.

KroMignon

@hbatalha said in Working with Qthreads:

Can you provide some example code? I have a basic idea one what you are saying but not enough to translate it to code.

Qt framework is an asynchronous framework, this helps for example for UI to avoid locking the main thread and enable user interactions.

One way to do it would be to create a list parameters for the process you want to start and go through this list to start each process at the end of previous one.

For example:

struct ProcInfo
{
    QString procName;
    QStringList procParameters;
    // add what ever you need
};

Then use this to start each process after the other:

class Launcher : public QObject
{
    Q_OBJECT

   QProcess mProc;
   QList<ProcInfo> mProcToStart; 
public:
   Launcher(QObject * parent = nullptr) : QObject(parent)
   {
       mProc.setProcessChannelMode(QProcess::MergedChannels);
       connect(&mProc, &QProcess:stateChanged, [this](QProcess::ProcessState newState) {
           if(newState == QProcess::NotRunning)
               startNext();
           });
   }

   void startProcs(const QList<ProcInfo> &toStart)
   {
       mProcToStart = toStart;
       startNext();
   }
   void startNext()
   {
        if(mProcToStart .isEmpty())
            return;
        auto nextP = mProcToStart.takeFirst();
        mProc.start(nextP.procName, nextP.procParameters);
   }
};

Something like that, up to you to finish it ;)

JKSH

@hbatalha said in Working with Qthreads:

@JonB said in Working with Qthreads:

Like I said, use the signal QProcess::finished. When one sends that, start a new one. That of course makes your processes execute sequentially, instead of at the same time.

Can you provide some example code? I have a basic idea one what you are saying but not enough to translate it to code.

First, get familiarized with the concept of signals and slots: https://doc.qt.io/qt-5/signalsandslots.html This is a core part of most Qt applications. When you know how to use signals and slots, many things will become clearer.

hbatalha

@KroMignon said in Working with Qthreads:

Then use this to start each process after the other:

Is it possible to run multiple processes at the same time.

Also, I couldn't get the code to work, maybe I am missing something or I am using it wrong:

    QList<ProcInfo> procs;
    Launcher l;
    for(auto const& e: selections)
    {
        QStringList args;

        args << "-some_args" << e;

        procs.push_back({"program.exe", args});
    }

    l.startProcs(procs);

Christian Ehrlicher

@hbatalha said in Working with Qthreads:

Is it possible to run multiple processes at the same time.

As I already said - simply create more than one QProcess instance.

hbatalha

@JKSH said in Working with Qthreads:

First, get familiarized with the concept of signals and slots: https://doc.qt.io/qt-5/signalsandslots.html This is a core part of most Qt applications. When you know how to use signals and slots, many things will become clearer.

I did it and was able to do this, which works(is it correct?)

void Dialog::on_pushButton_3_clicked()
{
    for(int i = 0; i < 3; ++i)
    {
        QStringList args;

        args << "some_args" << selections.takeFirst();

        QProcess* pro = new QProcess;

        pro->setProcessChannelMode(QProcess::MergedChannels);

        connect(pro, &QProcess::stateChanged, [this](QProcess::ProcessState newState)
        {
            if(newState == QProcess::NotRunning)
                start_next_process();
        });

        pro->start("program.exe", args);
    }
}

void Dialog::start_next_process()
{
    if(!selections.isEmpty())
    {
        QStringList args;

        args << "some_args" << selections.takeFirst();

        QProcess *process = new QProcess;

        connect(process, &QProcess::stateChanged, [this](QProcess::ProcessState newState)
        {
            if(newState == QProcess::NotRunning)
                start_next_process();
        });

        process->setProcessChannelMode(QProcess::MergedChannels);

        process->start("prgram.exe", args);
    }
}

I took at look at this but haven't yet completely understood

JonB

@hbatalha
You are working from @KroMignon's template. You will not need the QProcess::finished signal I mentioned/your link now refers to because he is effectively doing this (indirectly) via the QProcess::stateChanged signal he connects instead. Which is fine.

Have a look again at his code. In yours you have effectively copied the same block of code out of start_next_process() and into your on_pushButton_3_clicked(). You do not need to do that (repeating the same code is always a suspicious sign). His code is designed for you to use a selections member variable to "queue up" the 3 (in your case) processes you'd like it to run. Your code does no mesh well with that. Please take the time to understand how @KroMignon's approach works.

KroMignon

This post is deleted!