Concurrent map equivalent

Defohin

Hi guys, there's a class called ThreadPoolExecutor on Python that has a function called map, that function accepts a method to be called, and iterables to be passed to the function, for example:

def example(self, name, foo, bar):
    ....

# 10 is the total of workers
with ThreadPoolExecutor(10) as executor:
    executor.map(
        example,
        self.list_of_names,
        itertools.repeat(foo, len(self.list_of_names)),
        itertools.repeat(bar, len(self.list_of_names))
    )

Imagine that list_of_names is a list of names, imagine that foo and bar has a constant value, for example, foo = hello and bar = world, we need an iterable to be passed to the map function, then the itertools.repeat will create an iterable repeating the same thing the same size of list_of_names. In the end, example will have as argument each name from list_of_names, foo and bar.

I was wondering, how to simulate this on Qt? How to create a concurrent call of a method from a QMap<QString, QString> and also pass two additional QString as argument?

mrjj

Hi

Imagine that list_of_names is a list of names

Yes that totally works for me ;)

It sounds high level and nice java class,
not sure Qtt can match that 100% but it seems in function the same
as
http://doc.qt.io/qt-5/qtconcurrent-index.html
with
http://doc.qt.io/qt-5/qtconcurrent.html#map

But I didnt see a way to add extra parameters :(

Defohin

I tried that way, but it crashed:

QStringList names = { "john", "jane" };
QString extra = "doe";

auto example = [=] (const QString &name) {
    qDebug() << name << extra;
};

QtConcurrent::map(names, example);

I thought about using foreach and QtConcurrent::run but I will have to call waitForFinished.

QStringList names = { "john", "jane" };
QString extra = "doe";

auto example = [=] (const QString &name, const QString &extra) {
    qDebug() << name << extra;
};

foreach (QString name, names) {
    QFuture<void> result = QtConcurrent::run(example, name, extra);
    result.waitForFinished();
}

I don't know if the way I made can lead me to problems, a problem I can see already is: How can I pause them if it's a loop? For instance...

What you think about that?

mrjj

Hmm that was odd. does look ok.
will names survive the scope?

Well the foreach is not bad but dont
result.waitForFinished();

block the GUI thread ?

Defohin

@mrjj What you mean with names surviving the scope? I don't know what happens, it just crashes.
The second option seems to be more "the right way" cause I'm passing the values as argument not capturing it in a lambda, also I can create a method and call, the first option I can't call a method.
I tried this:

auto example = [=] (const QString &name, const QString &extra) {
    for (int i = 0; i < 9999; ++i) {
        qDebug() << name << extra;
    }
};

In the second option to see if it blocks the gui thread but I'm able to move it, I don't know if I'm testing the right way.

edit

I just realized that it does block the gui thread, I incremented the 9999 to 9999999 and well... nothing shows.

What can I do?

kshegunov

@Defohin said in Concurrent map equivalent:

What you mean with names surviving the scope?

He means this from the docs for QtConcurrent::map:

Calls function once for each item in sequence. The function is passed a reference to the item, so that any modifications done to the item will appear in sequence.

Once you leave the scope of the function, the reference you'd passed is invalid and the program crashes. Perhaps you need mapped() instead?

mrjj

Hi
I think concurrent will be more fun as it dont block the GUI thread.
as in.

void mapFunction(int& n) {
qDebug() << n;
}

void test() {
  QList<int>* vectorOfInts =  new QList<int>;
  for (int i = 0; i < 1000; i++)
    vectorOfInts->push_back(i);

  QFuture<void> future = QtConcurrent::map(*vectorOfInts, mapFunction);

} // end of scope

Defohin

@kshegunov I tried using mapped but it doesn't work with lambdas.

I removed the waitForFinished and it's working... but I don't know if something bad will happen by doing that...

Defohin

@mrjj Your way works fine but I can't pass additional parameters. The second approach is not blocking anymore cause I removed the waitForFinished but I don't know if it's going to be a bad thing...

Defohin

I just realized something... if though it's working, if I close the window it will continue running...

What can I do?

mrjj

@Defohin
Hi , i think you can cancel via
http://doc.qt.io/qt-5/qfuture.html#cancel
http://doc.qt.io/qt-5/qfuturewatcher.html#cancel

I think you need to use QFutureWatcher if u dont use waitForFinished.
All credits to @kshegunov :)

Defohin

@mrjj

Be aware that not all asynchronous computations can be canceled. For example, the QFuture returned by QtConcurrent::run() cannot be canceled; but the QFuture returned by QtConcurrent::mappedReduced() can.

:(

mrjj

@Defohin
Oh. not all can be canceled. :(
Well if u need to be able to stop at random then
threads might be the best solution.

Defohin

I don't want the pause, continue or the stop function, I just want to run a list of string over a method and pass additional arguments to it in parallel.
Using QtConcurrent::map won't work cause I can't pass additional arguments.
Using QtConcurrent::run won't work cause I'll need to call waitForFinished and it will block the GUI thread... (QFutureWatcher::cancel or QFuture::cancel won't work either cause it's not possible to cancel a QFuture returned by QtConcurrent::run)...

What can I do? I'm out of ideas.

Devopia53

@Defohin

like this:

QStringList names = { "john", "jane" }; // global or member variable

void YourClass::startJobs()
{
    QString extra = "doe";

    auto example = [extra] (QString &name) {
        qDebug() << name << extra;
    };

    QFuture<void> future = QtConcurrent::map(names, example);
}

mrjj

Hi
If cancel is a MUST have, then QThreads seems the only option :(

Defohin

@Devopia53 It's still crashing for me.
@mrjj I don't want to use cancel, as i said, I just want to run in parallel the method for each item of a string list and pass a few additional arguments. No need to cancel.

mrjj

@Defohin said in Concurrent map equivalent:

No need to cancel.

By cancel i mean to terminate before its finished. Like closing program.
In that regards you need "cancel" :)

Defohin

@mrjj Yes, if the application is closed I want to finish the threads as well. I thought it was a default behavior.

Can you give me an example of what I can do? I'm trying to use QRunnable, but I have no idea if it's going to work.

VRonin

@Defohin said in Concurrent map equivalent:

Using QtConcurrent::map won't work cause I can't pass additional arguments.

Just use a function object

struct MapHelper{
MapHelper(){}
QString m_additionalArgument1;
QString m_additionalArgument2;
 typedef QString result_type;
    QString operator()(const QString &val)
    {
        return m_additionalArgument1+val+additionalArgument2;
    }
}

MapHelper helper; // TODO: make sure to manage its lifecycle
helper.m_additionalArgument1 = "Prefix ";
helper.m_additionalArgument2 = " Suffix";
QtConcurrent::map(list_of_names,helper);