Solved How to do a heavy consuming task in background and limit its CPU usage?
-
@mbruel said in How to do a heavy consuming task in background and limit its CPU usage?:
I've noticed the loading of the second Tree could be delayed, it is not so important for the main usage of the app so I've put it in a std::thread.
What's the fascination with the STL I will never understand ... Run your code through a regular
QThread
and batch the work with the help of a timer.Is there another way than adding some sleeps in my task thread to have the GUI responsive?
Sleep (and variants of it) is the most notorious way to pause execution and you should avoid it like the plague. Sleep can't be interrupted, and can't be reliably controlled, it's a bad design! If you really need to have a timed wait, then use a semaphore.
Is there a QT way to launch a background task that would be limited on CPU usage so the GUI stays responsive?
The Qt way is not to use STL but to use Qt. Use the (
QThread
's) event loop to batch up your work.Kind regards.
-
@kshegunov
haha yeah I guess it is a way to go...
I've just used std::thread cause it tooked me less than 1min to add a rebuild...
basically I wouldn't mind to have thread::sleep not interrupt-able as the sleep would be like 100ms (the interruption would be caught quite fast just after) and that my task is at loading...But anyway, the app is in QT so I'm happy to change it full QT. Can you advise me on a proper way to do this.
Basically, in my MainWindow::loadTrees I've this _tree1->load(), _tree2->load(true)
_tree1 and _tree2 being instances of TreeModel that inherit from QStandardItemModel.The TreeModel::load function has a test on a boolean to differentiate the 2 distinct types of Tree.
So I only want to use a Thread on one case (for _tree2)What I was doing with the STL way, would just have been to run _tree2->load(true) in a thread in MainWindow::loadTrees and add some thread::sleep in the load function in the corresponding case in TreeModel::load
MainWindow::loadTrees() { ... _backgroundThread = std::thread(&TreeModel::load, _tree2, true); _backgroundThread.detach();
void TreeModel::load(bool flat) if (flat) { ... for loops ... if (_isLoadingThreaded && ++numIter == _numberElementToLoadBeforeSleep) { std::this_thread::sleep_for (std::chrono::microseconds(_sleepTimeForBackgroundTask)); numIter = 0; }
Using QThread with QTimer means I need to add the Q_OBJECT macro in the header of TreeModel. Is there a way to not make it also inherit from QThread? As I said, I don't want my first case to be in another Thread...
Could I use QtConcurrent::run maybe for _tree2->load(true) and use a QTimer moved in that thread? -
I wouldn't bother with a sleep function. This will probably make it worse.
You could use a lower priority thread. It won't lighten the CPU usage (?) but it will automatically make other things more responsive. Not a great way to do it but it might help.
I don't know if you can redesign your application to load the different sections on demand or as needed. For example, you can load or create your top level tree items then load only a specific sub item if someone happens to expand it and you hadn't previously loaded the data for it. Some approach like this makes more sense then loading everything into memory especially if it is big (unless you have no choice for some reason).
-
@mbruel said in How to do a heavy consuming task in background and limit its CPU usage?:
basically I wouldn't mind to have thread::sleep not interrupt-able as the sleep would be like 100ms (the interruption would be caught quite fast just after) and that my task is at loading...
Okay, it's a valid argument. If I were you, however, I'd still go with a timer event instead.
Basically, in my MainWindow::loadTrees I've this _tree1->load(), _tree2->load(true)
_tree1 and _tree2 being instances of TreeModel that inherit from QStandardItemModel.
The TreeModel::load function has a test on a boolean to differentiate the 2 distinct types of Tree.
So I only want to use a Thread on one case (for _tree2)I'd do it with a loader object. Basically, you create a worker object that will load the tree. And when you need or want to you can push the object to a thread. E.g. (pseudocode):
class TreeModelLoader : public QObject { Q_OBJECT public slots: void load(TreeModel *); signals: void loaded(); // Emit this after the model was loaded };
and then:
TreeModel * model1, * model2; // The two models TreeModelLoader * loader = new TreeModelLoader(); loader->load(model1); // Load the first one from the current thread QThread * thread = new QThread(); QObject::connect(loader, &TreeModelLoader::loaded, thread, &QThread::quit); QObject::connect(thread, &QThread::finished, loader, &QObject::deleteLater); QObject::connect(thread, &QThread::finished, thread, &QObject::deleteLater); thread->start(); // Start the thread loader->moveToThread(thread); // Move the worker // For this to work you need to register TreeModelLoader * with the meta-type system - qRegisterMetaType QMetaObject::invokeMethod(loader, "load", Q_ARG(TreeModelLoader * , model2));
For the cleanest solution however, you should declare a number of signals in your worker object (or
QThread
subclass, which is acceptable for this alternative approach) and connect those signals to slots in your model.Using QThread with QTimer means I need to add the Q_OBJECT macro in the header of TreeModel.
You should be doing this anyway,
QStandardItemModel
inherits fromQObject
so you should add theQ_OBJECT
macro for its subclasses.Is there a way to not make it also inherit from QThread? As I said, I don't want my first case to be in another Thread...
There is, and most of the time you won't need to subclass from
QThread
. See also this blog post for the long and better explanation.Could I use QtConcurrent::run maybe for _tree2->load(true) and use a QTimer moved in that thread?
In principle, but I don't see why you'd want to do it; you won't gain much by using a thread pool.
@Rondog said in How to do a heavy consuming task in background and limit its CPU usage?:
You could use a lower priority thread. It won't lighten the CPU usage (?) but it will automatically make other things more responsive. Not a great way to do it but it might help.
Not an option on Linux.
I don't know if you can redesign your application to load the different sections on demand or as needed.
It depends on the case, but this'd be the best solution here, I think.
Kind regards.
-
@Rondog said in How to do a heavy consuming task in background and limit its CPU usage?:
I wouldn't bother with a sleep function. This will probably make it worse.
Well this could have work but with the stl thread it won't because most processing then happen in the event loop which stays the one of the main thread...
You could use a lower priority thread. It won't lighten the CPU usage (?) but it will automatically make other things more responsive. Not a great way to do it but it might help.
There are no OS independent API and I'm not sure if it could work neither on Linux or Windows. I don't have enough time to investigate...
I don't know if you can redesign your application to load the different sections on demand or as needed. For example, you can load or create your top level tree items then load only a specific sub item if someone happens to expand it and you hadn't previously loaded the data for it. Some approach like this makes more sense then loading everything into memory especially if it is big (unless you have no choice for some reason).
Well I'm working on an extreme case that not many users will use but a few. The problem is that one root item of my tree can have 300k items under it at the first level. This is the process that is slow, I'm not sure why.... So delegating later could be done but there could be 1min wait when we try to expand this item...
That's why I would like to find an option were I could spread all the loading in background if I could make sure that it won't keep the CPU and block the main thread... It's a shame we don't have such control for background tasks... -
@kshegunov
I've updated my code to use a Loader that will be the worker in a Thread for the loading function of my Tree.
I've put a break of 2s every 1000 items using QThread::currentThread()->sleep(2).
This way on my laptop, the thread is using between 5 and 15% of the CPU.
It would be nice if we could be hardware independent and so if we could fix the usage of the CPU used by a Thread (like by scheduling it less often) but I guess I will go for this solution...
I'll just warn the user it is still loading and add a button to stop using the sleep and finish the job full process if needed.
Thanks for your help ;) -
What if instead of 1 thread scanning all of it you use 1 thread to scan each item. something like a QtConcurrent::map call (but done using QThreads and checking not to start too may threads all at once). this should be relatively manageable as you can control how intensive on the CPU it is based on how many threads at once you send
-
@mbruel said in How to do a heavy consuming task in background and limit its CPU usage?:
I've put a break of 2s every 1000 items using QThread::currentThread()->sleep(2).
Instead of sleeping you can just post a timed event, so your thread will (also) stay responsive. For example:
QTimer::singleShot(2000, this, &MyLoaderClass::slotThatLoads1000Items);
It would be nice if we could be hardware independent and so if we could fix the usage of the CPU used by a Thread (like by scheduling it less often)
This is the OS's scheduler job, and sadly not all schedulers were written equal. As mentioned Linux's won't care for thread priorities (Windows supports it) so sometimes you may need to walk around this, just bear in mind it's an OS deficiency.
Thanks for your help ;)
You're welcome.
-
@VRonin said in How to do a heavy consuming task in background and limit its CPU usage?:
What if instead of 1 thread scanning all of it you use 1 thread to scan each item. something like a QtConcurrent::map call (but done using QThreads and checking not to start too may threads all at once). this should be relatively manageable as you can control how intensive on the CPU it is based on how many threads at once you send
I don't really see what you mean... If I use several threads to do the job they will all work at 100% CPU during their scheduling if I don't put any sleep. So the overall would be the same. Or am I missing something?
-
@mbruel let's say you have a tree with 100 nodes and each of them has 50 sub nodes.
One thread will check the first node and then terminate, a new thread is then started to check the first sub node and then terminate, then another thread will do the second sub node and so on.
You can start, say,qMax(1,QThread::idealThreadCount()/2)
threads at once so you should you'd use at most half of your cores