Concurrent map equivalent



  • Hi guys, there's a class called ThreadPoolExecutor on Python that has a function called map, that function accepts a method to be called, and iterables to be passed to the function, for example:

    def example(self, name, foo, bar):
        ....
    
    # 10 is the total of workers
    with ThreadPoolExecutor(10) as executor:
        executor.map(
            example,
            self.list_of_names,
            itertools.repeat(foo, len(self.list_of_names)),
            itertools.repeat(bar, len(self.list_of_names))
        )
    

    Imagine that list_of_names is a list of names, imagine that foo and bar has a constant value, for example, foo = hello and bar = world, we need an iterable to be passed to the map function, then the itertools.repeat will create an iterable repeating the same thing the same size of list_of_names. In the end, example will have as argument each name from list_of_names, foo and bar.

    I was wondering, how to simulate this on Qt? How to create a concurrent call of a method from a QMap<QString, QString> and also pass two additional QString as argument?


  • Qt Champions 2016

    Hi

    Imagine that list_of_names is a list of names

    Yes that totally works for me ;)

    It sounds high level and nice java class,
    not sure Qtt can match that 100% but it seems in function the same
    as
    http://doc.qt.io/qt-5/qtconcurrent-index.html
    with
    http://doc.qt.io/qt-5/qtconcurrent.html#map

    But I didnt see a way to add extra parameters :(



  • I tried that way, but it crashed:

    QStringList names = { "john", "jane" };
    QString extra = "doe";
    
    auto example = [=] (const QString &name) {
        qDebug() << name << extra;
    };
    
    QtConcurrent::map(names, example);
    

    I thought about using foreach and QtConcurrent::run but I will have to call waitForFinished.

    QStringList names = { "john", "jane" };
    QString extra = "doe";
    
    auto example = [=] (const QString &name, const QString &extra) {
        qDebug() << name << extra;
    };
    
    foreach (QString name, names) {
        QFuture<void> result = QtConcurrent::run(example, name, extra);
        result.waitForFinished();
    }
    

    I don't know if the way I made can lead me to problems, a problem I can see already is: How can I pause them if it's a loop? For instance...

    What you think about that?


  • Qt Champions 2016

    Hmm that was odd. does look ok.
    will names survive the scope?

    Well the foreach is not bad but dont
    result.waitForFinished();

    block the GUI thread ?



  • @mrjj What you mean with names surviving the scope? I don't know what happens, it just crashes.
    The second option seems to be more "the right way" cause I'm passing the values as argument not capturing it in a lambda, also I can create a method and call, the first option I can't call a method.
    I tried this:

    auto example = [=] (const QString &name, const QString &extra) {
        for (int i = 0; i < 9999; ++i) {
            qDebug() << name << extra;
        }
    };
    

    In the second option to see if it blocks the gui thread but I'm able to move it, I don't know if I'm testing the right way.

    edit

    I just realized that it does block the gui thread, I incremented the 9999 to 9999999 and well... nothing shows.

    What can I do?


  • Qt Champions 2016

    @Defohin said in Concurrent map equivalent:

    What you mean with names surviving the scope?

    He means this from the docs for QtConcurrent::map:

    Calls function once for each item in sequence. The function is passed a reference to the item, so that any modifications done to the item will appear in sequence.

    Once you leave the scope of the function, the reference you'd passed is invalid and the program crashes. Perhaps you need mapped() instead?


  • Qt Champions 2016

    Hi
    I think concurrent will be more fun as it dont block the GUI thread.
    as in.

    void mapFunction(int& n) {
    qDebug() << n;
    }
    
    void test() {
      QList<int>* vectorOfInts =  new QList<int>;
      for (int i = 0; i < 1000; i++)
        vectorOfInts->push_back(i);
    
      QFuture<void> future = QtConcurrent::map(*vectorOfInts, mapFunction);
    
    } // end of scope
    


  • @kshegunov I tried using mapped but it doesn't work with lambdas.

    I removed the waitForFinished and it's working... but I don't know if something bad will happen by doing that...



  • @mrjj Your way works fine but I can't pass additional parameters. The second approach is not blocking anymore cause I removed the waitForFinished but I don't know if it's going to be a bad thing...



  • I just realized something... if though it's working, if I close the window it will continue running...

    What can I do?


  • Qt Champions 2016

    @Defohin
    Hi , i think you can cancel via
    http://doc.qt.io/qt-5/qfuture.html#cancel
    http://doc.qt.io/qt-5/qfuturewatcher.html#cancel

    I think you need to use QFutureWatcher if u dont use waitForFinished.
    All credits to @kshegunov :)



  • @mrjj

    Be aware that not all asynchronous computations can be canceled. For example, the QFuture returned by QtConcurrent::run() cannot be canceled; but the QFuture returned by QtConcurrent::mappedReduced() can.

    :(


  • Qt Champions 2016

    @Defohin
    Oh. not all can be canceled. :(
    Well if u need to be able to stop at random then
    threads might be the best solution.



  • I don't want the pause, continue or the stop function, I just want to run a list of string over a method and pass additional arguments to it in parallel.
    Using QtConcurrent::map won't work cause I can't pass additional arguments.
    Using QtConcurrent::run won't work cause I'll need to call waitForFinished and it will block the GUI thread... (QFutureWatcher::cancel or QFuture::cancel won't work either cause it's not possible to cancel a QFuture returned by QtConcurrent::run)...

    What can I do? I'm out of ideas.



  • @Defohin

    like this:

    QStringList names = { "john", "jane" }; // global or member variable
    
    void YourClass::startJobs()
    {
        QString extra = "doe";
    
        auto example = [extra] (QString &name) {
            qDebug() << name << extra;
        };
    
        QFuture<void> future = QtConcurrent::map(names, example);
    }
    
    

  • Qt Champions 2016

    Hi
    If cancel is a MUST have, then QThreads seems the only option :(



  • @Devopia53 It's still crashing for me.
    @mrjj I don't want to use cancel, as i said, I just want to run in parallel the method for each item of a string list and pass a few additional arguments. No need to cancel.


  • Qt Champions 2016

    @Defohin said in Concurrent map equivalent:

    No need to cancel.

    By cancel i mean to terminate before its finished. Like closing program.
    In that regards you need "cancel" :)



  • @mrjj Yes, if the application is closed I want to finish the threads as well. I thought it was a default behavior.

    Can you give me an example of what I can do? I'm trying to use QRunnable, but I have no idea if it's going to work.



  • @Defohin said in Concurrent map equivalent:

    Using QtConcurrent::map won't work cause I can't pass additional arguments.

    Just use a function object

    struct MapHelper{
    MapHelper(){}
    QString m_additionalArgument1;
    QString m_additionalArgument2;
     typedef QString result_type;
        QString operator()(const QString &val)
        {
            return m_additionalArgument1+val+additionalArgument2;
        }
    }
    
    MapHelper helper; // TODO: make sure to manage its lifecycle
    helper.m_additionalArgument1 = "Prefix ";
    helper.m_additionalArgument2 = " Suffix";
    QtConcurrent::map(list_of_names,helper);
    


  • @VRonin How to get the return from each one now?

    struct NameHelper
    {
        NameHelper(const QString &extra) : _extra(extra) { }
    
        typedef QString result_type;
    
        QString operator()(const QString &name)
        {
            return QString("Hello %1 %2").arg(name).arg(_extra);
        }
    
        QString _extra;
    };
    
    QStringList names = { "john", "jane" };
    QString extra = "doe";
    
    QFuture<QString> example = QtConcurrent::mapped(names, NameHelper(extra));
    

    example.result() is returning only "Hello john doe".



  • You can achieve the same with C++11 std::bind:

    QString concatenate(const QString& prefix,const QString& val,const QString& suffix){return prefix+val+suffix;}
    QtConcurrent::mapped(list_of_names,std::bind(concatenate,"Prefix ",std::placeholders::_1," Suffix"));
    

    How to get the return from each one now?
    [...]
    example.result() is returning only "Hello john doe".

    use results() instead of result()



  • It's working nicely, but one last question; Is it GUI blocking? I tried to use a for loop inside of the function and it's now showing anything until it's finished.



  • @Defohin said in Concurrent map equivalent:

    Is it GUI blocking?

    no, unless you force it to be

    I tried to use a for loop inside of the function

    could you post the code?



  • struct NameHelper
    {
        NameHelper(const QString &extra) : _extra(extra) { }
    
        typedef QString result_type;
    
        QString operator()(const QString &name)
        {
            for(int i = 0; i < 999999; ++i) {
                qDebug() << "block";
            }
    
            return QString("Hello %1 %2").arg(name).arg(_extra);
        }
    
        QString _extra;
    };
    
    QStringList names = { "john", "jane" };
    QString extra = "doe";
    
    QFuture<QString> example = QtConcurrent::mapped(names, NameHelper(extra));


  • operator() will be executed in another thread, it will not block the GUI thread, it will not return until completed (of course)



  • @VRonin But why is it not appearing the window If it's running in another thread?



  • @Defohin said in Concurrent map equivalent:

    But why is it not appearing the window If it's running in another thread?

    WOW! we are taking this on a whole new level here! you need a QFutureWatcher on the QFuture and a slot connected to the resultReadyAt signal to display the results in the GUI



  • You are a god, thank you.



  • I definitely am not, trust me.

    Just as a final remark, in your case, if you can use C++11, I'd use std::bind over the function object



  • I'm using std::bind now... I realized something:
    THe threads is not closing if the application closes, meaning that if I close the window it will still open. (if I do a heavy work inside the called function).



  • Since you are using mapped you can call cancel() to signal to abort the calculation



  • @VRonin Do I have to connect that to a signal from the window for when it's closing or something?



    • If you want to stop the calculation when the widget is closed reimplement the closeEvent and call cancel on the QFuture
    • If you want to stop the calculation when the widget is destroyed reimplement the destructor and call cancel on the QFuture


  • @VRonin Thank you very much.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.