QFutureWatcher - timing issue

[kshegunov]

@AnneRanch said in QFutureWatcher - timing issue:

The "spin" delay runs in its own thread - which is not necessary since the delay is cumulative

In its own threads.

- no need for parallel processing.

Then why do you explicitly use parallel processing? ... then wonder why the time is approximately scaled by the number of pooled threads ... this is all very confusing ...

BUT the dialog.exec() "uses" all four CPU cores ( as noted in the progress dialog text ) - hence four threads .

QDialog::exec doesn't use any cores at all. Your QtConcurrent::map does.

Keeping in mind that I really do not have means monitoring the dialog.exec()
iterations.

There are no iterations. QDialog::exec waits for an event to appear in the event queue, if there's none it just sleeps until one happens to appear (be it a GUI event or a signal/slot meta-call - a.k.a. queued connection call).

[Anonymous_Banned275]

@kshegunov said in QFutureWatcher - timing issue:

@AnneRanch said in QFutureWatcher - timing issue:

The "spin" delay runs in its own thread - which is not necessary since the delay is cumulative

In its own threads.

- no need for parallel processing.

Then why do you explicitly use parallel processing? ... then wonder why the time is approximately scaled by the number of pooled threads ... this is all very confusing ...

As pointed out many times - I am following a tutorial.
I have also said that most of the tutorial steps are not commented WHAT they do . Basically blindly following the tutorial - and the result is obvious.
Confusing to you , frustrating to me.

BUT the dialog.exec() "uses" all four CPU cores ( as noted in the progress dialog text ) - hence four threads .

Which does not make sense.

QDialog::exec doesn't use any cores at all. Your QtConcurrent::map does.

Keeping in mind that I really do not have means monitoring the dialog.exec()
iterations.

I do not like to get sidteracked by niticking on sentic.
The QProgresDialog ( n tutorial ) outputs

dialog.setLabelText(QString("Progressing using %1 thread(s)...\nTODO Elapsed time %2  [mS] ")
                    .arg(QThread::idealThreadCount()).arg(timer.elapsed()));

So why should I question WHO is using the threads - dialog or QtConcurrent?

How does that leads to solution ?
I am not complaining about your comment, just trying to maintain some priorities in logic to find the problem.

> There are no iterations. QDialog::exec waits for an event to appear in the event queue, if there's none it just sleeps until one happens to appear (be it a GUI event or a signal/slot meta-call - a.k.a. queued connection call).

**OK THIS IS VERY GOOD POINT and IMPORTANT,

My test delay process is initialized by standard button push...
"" can somebody describe WHAT EVENT actually starts QProgressDialog::exec ?**

I am under the impression that QProgressDialog constructor does that.
Am I wrong ?

And HOW does the delay process continue from starting point ?

Secondly
so what are "iterations" for ?
They are passed to QProgressDialog constructor...

Challenge to group
Would somebody some up with old fashioned flow chart to put the process
from "button pushed" to "delay completed " ?
My verbal output track sort of does it , but obviously needs better explanations what each step does.

New crazy thought - the original tutorial code worked (!) but it used default
constructor - without any parameters passed to the class and without setting any parameters in code - just used all unpublished defaults.
With one of the defaults , forgot which one, the actual progress bar supposedly should show up AFTER 4 second delay .

In current failure mode the progress bar shows 10% (immediately) then shows 50% ( after about 3 seconds) - the percentage is exactly "4 seconds " between the updates when iteration is set to 10 !

[kshegunov]

@AnneRanch said in QFutureWatcher - timing issue:

As pointed out many times - I am following a tutorial.
I have also said that most of the tutorial steps are not commented WHAT they do . Basically blindly following the tutorial - and the result is obvious.

Then read in docs what they do so you understand it, at least that's my advice. QtConcurrent::map isn't an alien we need a team of scientists to talk to, right? Just look it up, that's what I do when I follow tutorials.

Confusing to you , frustrating to me.

Yes, yes, frustration leads to anger, anger leads to the dark side and all that. Less emotion, more practicality helps, I've found out.

I do not like to get sidteracked by niticking on sentic.

Actually you do, you just don't realize it. You have an unknown machine you're trying to figure out, but you claim you're not interested in how the cogs it's made of fit together. Should I mention such an approach is utterly futile?

The QProgresDialog ( n tutorial ) outputs

dialog.setLabelText(QString("Progressing using %1 thread(s)...\nTODO Elapsed time %2  [mS] ")
                    .arg(QThread::idealThreadCount()).arg(timer.elapsed()));

So? This is evaluated before the dialog is even shown to the screen and has nothing to do with the dialog itself.

So why shoudI question WHO is using the threads - dialog or QtConcurren ?

It should, so you know where the problem lies. If you don't understand where the error is, then you're not going to be able to fix it, ever.

How does that leads to solution?

I don't know exactly what kind of solution you expect as I can't seem to fathom the problem. You split and process some vector in several threads, I get that. But I don't get what times and delays and elapsed timers have to do with it ...

For reference, what's the output of the following snippet:

QObject::connect(&futureWatcher, &decltype(futureWatcher)::progressRangeChanged, [] (int from, int to) -> void  {
    qDebug() << "Map from " << from << " to " << to;
});
QObject::connect(&futureWatcher, &decltype(futureWatcher)::progressValueChanged, [] (int value) -> void  {
    qDebug() << "Map progress: " << value;
});

[Anonymous_Banned275]

Assuming the QprogressDialog is advanced / controlled by QFutureWacher SIGNALs of :"value" and "range"

I have added means of monitoring these signals

 // create test SLOT to monitor progressValueChanged(int)

qDebug() << "create test SLOT to monitor progressValueChanged(int)";
QObject::connect(&futureWatcher, SIGNAL(progressValueChanged(int)), this, SLOT(on_doTaskButton_2_clicked));
QObject::connect(&futureWatcher, SIGNAL(progressRangeChanged(int,int)),this, SLOT(on_doTaskButton_3_clicked)));

The result output is here

 Start the computation - 
apply delay function spin to each vector
 RUN dialog.exec() Display the dialog and start the dialog event loop
 START dialog.exec() 
futureWatcher.progressValue()  0
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  0
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  1
Estimated single iteration delay elapsed time  1001  mS
iteration #  0 in thread 0x7f380b7fe700
Estimated single iteration delay elapsed time  1001  mS
iteration #  1 in thread 0x7f37fbfff700
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  2
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  3
Estimated single iteration delay elapsed time  1001  mS
iteration #  2 in thread 0x7f3808eea700
Estimated single iteration delay elapsed time  1008  mS
iteration #  3 in thread 0x7f37fb7fe700
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  4
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  5
Estimated single iteration delay elapsed time  1001  mS
iteration #  5 in thread 0x7f37fbfff700
Estimated single iteration delay elapsed time  1008  mS
iteration #  4 in thread 0x7f380b7fe700
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  6
Estimated single iteration delay elapsed time  1001  mS
iteration #  6 in thread 0x7f3808eea700
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  7
Estimated single iteration delay elapsed time  1001  mS
iteration #  7 in thread 0x7f37fb7fe700
Estimated single iteration delay elapsed time  1001  mS
iteration #  8 in thread 0x7f37fbfff700
Estimated single iteration delay elapsed time  1001  mS
iteration #  9 in thread 0x7f380b7fe700
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  8
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  9
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  10
QDEBUG TRACE track  SIGNAL SLOT
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_2_clicked
@line     512
 SLOT value access  11
futureWatcher.progressValue()  10
 END dialog.exec()  3030  mS

Conclusion
the SIGNAL monitoring the value is changed 11 times - pretty much what is expected. I'll add showing the actual "value" at each time int changes - just to further verify it - it should be 10, 20, 30 etc.

The range does not change and it should not - should remain 0 thru 100.

[kshegunov]

@AnneRanch said in QFutureWatcher - timing issue:

The range does not change and it should not - should remain 0 thru 100.

I somehow doubt that assertion, more specifically the numbers you cite, and the conclusions you draw from it. This is why I requested the output from the snippet in my previous post.

[Anonymous_Banned275]

"Programs written with QtConcurrent automatically adjust the number of threads used according to the number of processor cores available. "

Hence there is no single usage of QFuture vectors - but FOUR of them are used running in four CPU cores at once.
Hence QProgressDialog "runs out" the correctly timed iterations FOUR times faster.
Thus the total desired time delay is incorrect.

[Pl45m4]

@AnneRanch said in QFutureWatcher - timing issue:

FOUR times faster.

Four threads won't increase the processing time of your whole task by a factor of four (if they share the same data)

Might be interesting:
(esp. the Concurrent part)

• https://www.logicbig.com/quick-info/programming/multi-threading.html

[Anonymous_Banned275]

@Pl45m4 said in QFutureWatcher - timing issue:

@AnneRanch said in QFutureWatcher - timing issue:

FOUR times faster.

Four threads won't increase the processing time of your whole task by a factor of four (if they share the same data)

Might be interesting:
(esp. the Concurrent part)

• https://www.logicbig.com/quick-info/programming/multi-threading.html

Hence there is no single usage of QFuture vectors - but FOUR of them are used running in four CPU cores at once.
Hence QProgressDialog "runs out" the correctly timed iterations FOUR times faster.
Thus the total desired time delay is incorrect.

[SGaist]

@AnneRanch said in QFutureWatcher - timing issue:

Hence there is no single usage of QFuture vectors - but FOUR of them are used running in four CPU cores at once.
Hence QProgressDialog "runs out" the correctly timed iterations FOUR times faster.
Thus the total desired time delay is incorrect.

You are using the wrong assumption that the four cores of your machine are each running exclusively one of these tasks.

They are not. Your processor cores are shared and runs variety of tasks from which your application and its threads.

Just take a look at your platform system analyser. You'll see that there's a lot happening under the hood.

[kshegunov]

@AnneRanch said in QFutureWatcher - timing issue:

Hence there is no single usage of QFuture vectors - but FOUR of them are used running in four CPU cores at once.

Hm, very weird, if only someone had mentioned this before ...!

Hence QProgressDialog "runs out" the correctly timed iterations FOUR times faster.

Approximately. Which is the whole point of running anything in parallel.

Thus the total desired time delay is incorrect.

There's no such thing, I believe I already said that on more than one occasion. GUI programming is event driven, there's no specific time for waiting, or specific order in which things happen; you get events, you respond to events, this is all. If it takes 3 seconds or 5, or 10 seconds or an hour is beside the point. When the event/signal is sent/emitted, then you process it and this is all ...

@SGaist said in QFutureWatcher - timing issue:

You are using the wrong assumption that the four cores of your machine are each running exclusively one of these tasks.

It's possible, albeit rather rare.

@Pl45m4 said in QFutureWatcher - timing issue:

Four threads won't increase the processing time of your whole task by a factor of four (if they share the same data)

Actually, they should. They usually don't as per Amdahl's law. I've a piece I'd written for work that scales almost perfectly with the number of cores, however the task I'd implemented is such that it is extremely parallelizable.

[Pl45m4]

@kshegunov said in QFutureWatcher - timing issue:

Actually, they should. They usually don't as per Amdahl's law. I've a piece I'd written for work that scales almost perfectly with the number of cores, however the task I'd implemented is such that it is extremely parallelizable.

If you use threads on all your processor cores, at least some of them have other tasks to do (OS background services etc.), so you will always lose some msec to a few secs or am I wrong?

[Anonymous_Banned275]

@SGaist said in QFutureWatcher - timing issue:

@AnneRanch said in QFutureWatcher - timing issue:

Hence there is no single usage of QFuture vectors - but FOUR of them are used running in four CPU cores at once.
Hence QProgressDialog "runs out" the correctly timed iterations FOUR times faster.
Thus the total desired time delay is incorrect.

You are using the wrong assumption that the four cores of your machine are each running exclusively one of these tasks.

They are not. Your processor cores are shared and runs variety of tasks from which your application and its threads.

Just take a look at your platform system analyser. You'll see that there's a lot happening under the hood.

My "assumption " is based on debug output which outputs 4 calls to "spin" in one shot. Then it pauses (1second) and does another 4 calls , pauses and output 2 calls - thus exhausting the 10 size vector array , set for 1 second each, in 3 seconds.
I could add another emit SIGNAL at 1 second intervals and output debug time marks at each second, but I do not need any further proves for myself.

BTW - recall my remark about Peanut gallery ?
From start I was after to find out why the OVERALL timing discrepancy and now I am getting all kinds of "advises" . For example - why I cannot have precision in mS . I do not need "precision". But that seem to be trend in some discussions - sidetrack to irrelevant issue instead of staying on main task.
(And you call my opinion rude? Yes, I am being rude because my only defense is to ignore these superficial and sometime repetitious posts. What else can I do ?)
Posters have "right" to opinions, but I am " being rude" when I state mine in my language style. I am no longer referring to your remark in this post - this is my general feeling about how it works 'lopsided" in this forum.
As long as I agree with stated opinion it is fine, if not then IBTL ...

.

[kshegunov]

@Pl45m4 said in QFutureWatcher - timing issue:

If you use threads on all your processor cores, at least some of them have other tasks to do (OS background services etc.), so you will always lose some msec to a few secs or am I wrong?

Sure, the quantity of such overhead varies depending on the setup/usage, though. In the specific case I was mentioning, the piece of code was run on a cluster, specifically for the purposes of numbercrunching, so there's only the bare minimum of cpu time consumers. Also notably, it depends heavily on the problem itself. Some problems lend themselves well to computing in parallel, while others require you to have either memory fences, or serializations that kill the "ideal" scalability.

@AnneRanch said in QFutureWatcher - timing issue:

I could add another emit SIGNAL at 1 second intervals and output debug time marks at each second, but I do not need any further proves for myself.

Proofs of? That QtConcurrent::map does exactly what's documented?

Posters have "right" to opinions, but I am " being rude" when I state mine in my language style. I am no longer referring to your remark in this post - this is my general feeling about how it works 'lopsided" in this forum.
As long as I agree with stated opinion it is fine, if not then IBTL ...

Everybody has a right to an opinion, this doesn't mean others are obliged to respect said opinion just because it is (i.e. exists), though. And just to add some context, opinions are opinions, but a statement of fact isn't an opinion; things work in a specific way regardless of your opinion, or mine.

As a physicist I'd allow myself the brazenness to give you some food for thought (an observation if you will) - you seem to be one of these people that believe that the universe should conform to their expectations, and if it doesn't then it is wrong. However, the real world doesn't operate like this and you'd be doing yourself a favor to cure yourself of such aspirations; you'd have a much more productive and happier time.

[Anonymous_Banned275]

SUCCESS !

Here is the tail of current debug output.
Works as expected , but needs some time tweaking.
The artificial 1 second "progress ticks" are done in about 7 seconds , while the real time consuming HCI scanner takes about 10 seconds to complete .

In closing I would like to thank to all who help by "holding my hand " and hugely contributed to success.

THANKS

 SLOT value access #  17
futureWatcher.progressValue()  20
 END dialog.exec()  7037  mS
TODO // set total delay to configuration dialog 
QDEBUG TRACE END of  time consuming task
file      ../CAT_V1/configuredialog.cpp
function  on_doTaskButton_clicked
@line     408
end timer TODO  
elapsed timer TODO  
success #num_res   2
success inquiry_info TODO   2
Elapsed real time  10238 [mS] 
 success nearby BT device  B8:27:EB:11:3F:82  ARM
success nearby BT device  00:15:83:15:A2:CB  d-SATA #1
code_text