Running into problems using a simple gui multithreaded application. FRUSTRATED AND NEED HELP !

[alonhalawi]

First of all, I must say, I am disappointed from the design of Qt.
I need to program a gui application that outer events change and update the UI.

My existing code is in python, which responsible of invoking the events to change the UI,
and I use PySide to combine the UI made by Qt with python.

AND NOW FOR THE PROBLEM

Basicly I have only two push buttons on the UI. One of the buttons starts a long hardware test, with 15 stages.

Now sync-ing the results with the UI is just a mess! cause as far as I checked, the UI can't be changed from outside of his own main class, can't even trigger a simple function to do it (already tried emitting an update func from another process). Not only that, the function updating the UI MUST end so the UI will update.

A simple comparison, I remember programming with Visual Basic (all versions, not only .net), and there was no problem what so ever updating the ui, so it will change Immediately. I repeat, the changes in VB happens just when you do them, they occure right on the time you change anything on the ui.

In summary, the only solution I got so far is another button (so the ui will gracefully respond to), to refresh the UI, somewhen when the user presses it. And by the way, I am positive that even that will not be smooth ..

Here are some parts of my code (assume that the tester product is called AA):

main.py
@

message queues

import multiprocessing as mp
mqAA2Gui = mp.Queue()
mqGui2AA = mp.Queue()

a simple way of sharing objects, I used a dictionary.

control = {'mqGui2AA':mqGui2AA, 'mqAA2Gui': mqAA2Gui, 'objGuiInstance':None}

creating a tester instance

AA_Instance=AA_Tester.MainAA(control)

creating a gui, passing it (by ref) the shared objects and AA_Instance

app = QApplication(sys.argv)
form = MainDialog(control=control, AA_Instance=AA_Instance)
form.show()
app.exec_()
@

@
class MainDialog(QDialog, pages5.Ui_Pages):

def init(self, parent = None, control = None, AA_Instance = None): # control
super(MainDialog, self).init(parent)

self.setupUi(self)
self.control = control

self.retry.released.connect(self.retry_call)
self.next.clicked.connect(self.next_clicked)
AA_Instance.emit_gui_update.connect(self.gui_update_handler)

self.init_AA_Process(parent, AA_Instance)

#####################################
def init_AA_Process(self, parent, AA_Instance):

self.AA = AA_Instance

self.process = mp.Process(target=self.AA.test)
print "mqGui2AA",self.control['mqGui2AA']
self.process.start()
@

class Ui_Pages in pages5.py
@
def retry_call(self):

print "retry_call is called",

print "page:", Ui_Pages.P_index

print "retry_call changing gui"
self.modify_pic(self.uut_dct_controls['/dev/ttyUSB0']['main'], './images/online_aa.jpg')
self.modify_text(self.uut_dct_controls['/dev/ttyUSB0']['status'], "serial/power not connected")
self.temp_sen_1_UUT_1.setStyleSheet("background-image: url(null);")
time.sleep(5)
@

@
def run_update(self):

item = None

print "in run_update"
time.sleep(0.3)

while self.control['mqAA2Gui'].qsize() > 0:
print "run_update(): waiting for mqAA2Gui queue"
item = self.control['mqAA2Gui'].get()
print 'run_update(): got item', item

if item == 'end':
break
else:
self.parse_gui_update(item)
else:
print "run_update(): no messages in mqAA2Gui, bailing out"
@

@
def next_clicked(self):
print "next_clicked is called",

print "page:", Ui_Pages.P_index

self.control['mqGui2AA'].put('next')

Ui_Pages.P_index = Ui_Pages.P_index % 6 + 1 # from 1 to 6
self.retranslateUi()
print "next_clicked is done"
@

[dheerendra]

Welcome to devnet. Your task is simple and you need to understand Qt better. When we compare there is always + and - with any tools/framework. For e.g Android does not do certain things the way Qt does. I can't squarely blame android/ios etc.

Coming your issue..

1. All UI objects are in main thread.
2. You create worker thread to your test/subtasks.
3. Communicate between main and worker thread using sigs/slot or events.

With this you will be able to achieve what you want.

You have to good understanding of how things work overall and make it.
Enjoy Qt.

[Chris Kawa]

Hi and welcome,
[quote author="alonhalawi" date="1404542847"]Now sync-ing the results with the UI is just a mess![/quote]Not if you do it right ;)
[quote author="alonhalawi" date="1404542847"]cause as far as I checked, the UI can't be changed from outside of his own main class[/quote]That's not true. Where did you get that?[quote author="alonhalawi" date="1404542847"]can't even trigger a simple function to do it[/quote]Qt definitely doesn't prohibit you from calling functions ;) You're doing something wrong
[quote author="alonhalawi" date="1404542847"](already tried emitting an update func from another process).[/quote] You don't emit a function. You emit a signal. Sounds like you're a little lost there.
[quote author="alonhalawi" date="1404542847"] Not only that, the function updating the UI MUST end so the UI will update.[/quote] Again - No it doesn't (but it should in a good design). The loop just has to have time to do repainting. There are ways to force the loop to process events even inside your methods, but I wouldn't suggest that (it's bad design).
[quote author="alonhalawi" date="1404542847"]A simple comparison, I remember programming with Visual Basic (...)[/quote]That's because this language is so simplified that it doesn't even grant you basic control over what is drawn when. It's a major shortcoming, not a feature. Imagine changing 1000 labels. VB will redraw them 1000 times weather you like it or not. With Qt you can draw them once, you can draw them 1000 times or draw them in some batches. It's your call. That's the beauty of it.

Now to the problem.
All UI updates should be done in a main thread. That is where the main event loop spins. You can create slots in your UI class that will update it (eg. setSomeValue(int value), setTime(QTimeDate time), updateLabel(QString text) etc.)
These should be small(probably one-liner) methods that just update a component and return.

Now the lengthy processing should be done in a separate thread. Whenever you want toupdate the state of UI you emit a signal defined in your processing class eg. emit someValueChanged(value), emit timeChanged(time), textChanged(text) etc.

The only thing to do now is to pair these classes so the slots get called when you emit a signall. You do it by calling connect for all the pairs. It's a very simple, efficient design, that grants you great control over what happens when.

In VB you would directly access UI controls from the processing thread which is just plain ugly and unnecessarily makes the two classes dependent. With the signal/slot approach UI doesn't even have to know there is any processing. It just reacts to signals. Similarly - worker thread doesn't know anything about UI(why should it?). It just lets the world know something happens within it by emitting signals.

"Here":http://pyqt.sourceforge.net/Docs/PyQt5/signals_slots.html#connecting-disconnecting-and-emitting-signals is a link to how you use signals/slots with PyQt, as you're clearly not doing it right.

[alonhalawi]

Here are a few things about the replies, tried to make it short and to the point, became a little long :-/ :

1. First of all, thanks for the reply. I'm kinda disapointed still ..

2. You're all right, I suppose I don't understand Qt too well, that's in the least ...

3. I'm sorry Qt fans, but as bad as it sounds, compared to VB, that simplification ( - any simplification) can be disadvantage.

But, the fact that the default is to be simplified and working makes it alot more attractive.

4. VB.net does let you control over what is drawn when (sure, not the default).

Moreover, the same element of simplification in VB.net,
may it be a disatvantage, is definitely not a shortcoming since you can be more specific and complex to control, as I said, what is beeing drawn when.

When you do that, it is alot more difficult, and then you can say to yourself, well dude, YOU ASKED FOR IT !!.

5. If it were my choice in the begin with, I'd go for python, and python only.

Not only is it simple, short and elegant, I'm usually given plenty of control over whatever I want.

6. I'm an embedded engineer, and I work mainly as a BareBoard Embedded, Linux Embedded and also Kernel sometimes.

I DO CARE ABOUT WHAT RUNS WHEN. Now, about the need to 'control', well, you can't expect to control EVERYTHING, that is given.

But in C for example, in the Linux/BareBoard env, Sync-ing surely becomes a big deal, so when I have to learn a whole new
set of env from scratch, and learn a bunch of Objects and Mechanism that actually takes its own preferance of control ..... well, you see why I'm frustrated ?

Sure there is a reason for everything, anything that's been done in Qt is for a good reason, though,
when I look at an object such as QObject/QThread/PrintEngine (if I remmember correctly, it was PrintEngine), and seeking what I want in the simplified and controlled version ...
I don't even know where to start.

(ALSO, You can see why I'm not all about Gui, not a big fan, I like printf/printk better ...)

7. Thanks for the help, I will look more carefully about the post given here and update about my problem. Hope to give a more positive answer next time/reply ..

[Chris Kawa]

The signal/slot mechanism is used throughout whole Qt library. In the scenario I described you get exactly the "instant update" you want. It's not harder or easier than more fine grained control. They're the same: signal, slot and a connection, nothing more. Possibly down to ~4 lines of code.

As for other languages. Python, to me, is a nice language for "small" things, an install script, a build tool, a tech demo when you need to check something quickly. You don't build Photoshop or an AAA game in Python. Just not that kind of performance and scale control.
VB, again - to me, is for people who don't want to be programmers yet they need to program something, like a data export from Excel etc. It's a little above Logo if you ask me :) Every time I need to use it I die a little inside :P It has its (limited) uses, but the world would do without it just as well if not better in my opinion.

printf/printk -there's a grungy feel to that sentiment and I understand that to a degree, but there's a reality check too - computers are not beige boxes that barely can run tetris anymore. they are so much much more and so are the interaction methods.

[alonhalawi]

I am working on other projects so it took me a while to go back to this one.

Chris Kawa, I agree with most of what you say, and I'll talk about those stuff later on another post ..

I tried emitting a signal from a process I created, to call a slot to change a picture in one of my controls on my main dialog. Still doesn't work. Other experiments do work, but again, none if them are what I desire.

Some little insights from what I tried.

1:
I see that the main dialog has an event loop, and it is a single threaded object. According to my experiments, if I click a button on the main dialog to change a picture (I use setStyleSheet,) it works. But again, it's not what I seek to do.

Basicly, the function does have to end if I do that, but I can call QApplication.processEvents() to force the chage before it ends. So, in a way, this is a function I might use. I bet this is not the common practice anyway ... (correct me if I'm wrong on this one)

Here is the code related to this insight:

2:
When I use the tester process to emit the signal to the gui process, the tester goes blocking, this is strange, only when the emitted slot call ends, the tester process continues. Now WHYYY does this phenomenon happends, those are not even the same process, aren't they !?

main.py
@
import threading, sys, Queue
import multiprocessing as mp
from PySide import *
from pagex import *

import AA_Tester

shared queue object between my threads

mqGui2AA = mp.Queue()

control dictionary

shared_objects = {'action': 'init', 'mqGui2AA':mqGui2AA}

AA_Instance=AA_Tester.MainAA(shared_objects)

qt gui

app = QtGui.QApplication(sys.argv)
form = MainDialog(shared_objects=shared_objects, AA_Instance=AA_Instance)
form.show()
app.exec_()
@

maindialog.py
@
class MainDialog(QtGui.QDialog, pages.Ui_Pages):
def init(self, parent = None, shared_objects = None, AA_Instance = None):
super(MainDialog, self).init(parent)
self.setupUi(self)

      self.next.clicked.connect(self.next_clicked)
      self.retry.released.connect(self.retry_call)

################ it spawns many threads, so I chosed to put the signal on the AA file #############
AA_Instance.emit_gui_update.connect(self.parse_gui_update)

      self.AA = AA_Instance
      self.process = mp.Process(target=self.AA.test)
      self.process.start()

@

pages.py
@
class Ui_Pages(object):
def next_clicked(self):
self.shr_objs['mqGui2AA'].put('next')

def retry_call(self):
self.modify_text( ... modifying text ... )
self.modify_pic( ... modifying pictures ... )

@

AATester.py
@
class MainAA(QtCore.QObject):

emit_gui_update = QtCore.Signal(list)

def init(self, shared_objects):

print "in MainAA::init"
QtCore.QObject.init(self, parent=None)

self.shr_objs = shared_objects

def test(self):

example of a test

while (True):
message = self.shr_objs["mqGui2AA"].get()
print "test(): page 1, got message",message

loading config

self.do_some_test()
#####################3 this should emit a gui update ################ it doesn't do it !!!!
ret = self.emit_gui_update.emit(['/dev/ttyUSB0', 1, 1])

if message == 'next':
break

@

I have removed most of the debug prints so you could see the code.

• • The first insite I wrote is just more frustrating, couse its a progress, and helpful ..

• I am not able to explain the second insite, according to all the posts here, this should not happend in Qt !!!*

[Chris Kawa]

My Python is a little rusty, but I believe this part is the problem:
@
AA_Instance.emit_gui_update.connect(self.parse_gui_update)
self.AA = AA_Instance
self.process = mp.Process(target=self.AA.test)
self.process.start()
@
It's not immediately obvious but for performance reasons there are different types of connect. This type is controlled by the type parameter in the connect() call.
The default value is AutoConnection that is evaluated to either DirectConnection or QueuedConnection (or others that are of no interest here).

When DirectConnection is used an emit is basically a simple function call. As every function call this is blocking.
When QueuedConnection is used emit puts a "marker" in the event loop of the thread that hosts the target object and the target thread calls the slot the next time it gets to process events. This is non-blocking and to be thread safe it uses mutexes (so it's a bit slower).

Now, another thing I said is "the thread that hosts the target object". In Qt each QObject is "assigned" to a specific thread (the main ui thread by default). Every slot is executed in the thread that object lives in, no matter from which thread the signal came.

The AutoConnection chooses the type of connection based on the threads the caller and receiver live in. Same thread -> direct, different -> queued.

In your case AA_Instance lives in the main thread. Now the tricky part is that a thread object itself also lives in the main thread, even though its run method (called by start) is started in the worker thread. Thread object is not in the same thread as the code it runs in start(). Because both - thread object and the receiver live in the same thread, Qt establishes a direct, blocking connection.

The usual way to deal with that is (in pseudo code)
@
UIObject ui = ...; //the receiver
QThread* t = ...; //a thread
Worker* w = ...; //a worker object
w->moveToThread(t); //assign the worker to the thread
connect(t, QThread::started, w, Worker::doSomeWork);
connect(w, Worker::someSignal, ui, UIObject::updateUi);
t->start(); //run the thread
@
Because the worker is moved to thread before the connect() the connection chosen is QueuedConnection.
Sorry for the C++'ish syntax. I know how to read Python but I'm terrible at writing it. I hope you can translate.

[alonhalawi]

Thank you Chris for the reply, I am trying it, I have already changed the connect param to QueuedConnection, but it didn't work.

I tried to do it with processes to make sure it won't be blocked, but it goes blocking even with processes.

the solution of moveToThread was tried before, and also didn't work, though I'm not sure I used the QueuedConnection attribute.

[alonhalawi]

Tried it, didn't work.

Now there must be some small detail I am missing -- again --.
But right now basically the thread doesn't start.

I really didn't want to go in to one of those tricky parts, that's why I chosed processes.

bq. In your case AA_Instance lives in the main thread. Now the tricky part is that a thread object itself also lives in the main thread, even though its run method (called by start) is started in the worker thread. Thread object is not in the same thread as the code it runs in start(). Because both – thread object and the receiver live in the same thread, Qt establishes a direct, blocking connection.

Now this is EXACTLY why I say, and you can and should quote me on this one, Qt TAKES ITS OWN PREFERENCE OF CONTROL !!

The whole idea of processes is that they all have their own domain of objects and whatever, and making something shared is simply shm get and attach, if you want it as direct as can be.

The whole idea of threads is to have a WHOLE DATA SEGMENT SHARED, but still, my responsibility to sync them.

The WHOLE IDEA OF 'MOVE TO THREAD' IS TO GET ME PISSED OFF !!

[Chris Kawa]

We should not mix processes with threads. They are completely different concepts. Processes know nothing about each other (unless using shared memory concepts - e.g. QSharedMemory). You can't connect objects in different processes.

Now as for threading - you easily loose your nerve, don't you? :) Threading is complicated and each language/lib has a different take on it. Qt's is that objects have thread affinity. Data is not that important, but access to it is (what thread reads/writes). To me it's quite elegant. You just mark an object as belonging to thread X and all its related code runs there. You still need to synchronize shared data access with mutexes and what not, don't get me wrong.

I guess Qt is just not for you if it makes you that mad. I've been using it for almost 7 years now and it's a love story :) Have you considered switching to a language/lib more to your liking?

[alonhalawi]

The reason of me loosing my nerve is that it supposed to be very simple.

If I had the ability to draw a rectangle on the screen and paint it according to anything, I have many ways to do it.

The only reason I chose Qt is for the graphics.

And it takes me a lot of digging, and getting to know the UI, the objects and the threading mechanisms, I mean, forcing me to read all of this stuff, then experiment, then I see what I have to define to make this simple stuff to work.

I'm positive I have missed something, but I am not going to port all my python work in PySide.

Look at the requirement for making this work:

1. manage signals and slots
   1.1 define slots for functions you want to invoke
   1.2 define signals (so you could emit the slots)
   1.3 connect the signal(s) to the slot(s)
   1.4 make sure it's a queued connection (it wouldn't work otherwise)
2. use a thread, not a process [1]
   2.1 use the moveToThread when creating
   2.2 use signal to emit changes on the gui (&make sure the func ends)
   2.3 use QtGui.QApplication.processEvents() to force the updates when the func doesn't end

I can go on ..

All this stuff to handle, none of this belongs to the code itself, only preparations ..

[1] BTW in c, syncing threads is with mutexes, and using globals, syncing process is different but they communicate with ipc e.g. mq, sockets, shm ..

How is any of this comfortable ?

[alonhalawi]

I bet something is wrong with this code:

main.py
@
AA_Instance=AA_Tester.MainAA(shared_objects)

app = QtGui.QApplication(sys.argv)
form = MainDialog(shared_objects=shared_objects, AA_Instance=AA_Instance)
form.show()
app.exec_()
@

maindialog.py
@
class MainDialog(QtGui.QDialog, pages5.Ui_Pages):

def init(self, parent = None, shared_objects, AA_Instance):

self.AA = AA_Instance

self.thread = QtCore.QThread()
self.AA.moveToThread(self.thread)
self.AA.run_test.connect(self.AA.test)
self.thread.start()
@

the thread doesn't start

[Chris Kawa]

[quote author="alonhalawi" date="1405262476"]
Look at the requirement for making this work: (...)[/quote]

1. Yes, you need to learn library to properly use library. What's the problem?
   1.1 one line
   1.2 one line
   1.3 one line
   1.4 no code (if you know the library)
2. If you're a programmer you should know the difference anyway. You can actually make it work with both. It's just different.
   2.1 QObject belongs to a thread it was created in. If you create the worker inside the running thread no need to use moveToThread, one line otherwise
   2.2 one line. What do you mean make sure it ends? It ends. It's a function call. If you mess up the setup and it directly calls lengthy slot then of course it doesn't end. What is surprising about that?
   2.3.No, you should not use processEvents. It's for cases when your code is so messed up you can't find other ways to fix it. It's a lifeboat, not a yaht. Should not happen in such a simple case. And again - a UI update should be a small function that ends right away. Don't do lengthy stuff in UI slots.

[quote]I can go on ..[/quote]Please do. (Denholm Reynholm ref. :) )

So, I counted 3-4 lines of preparation code (not counting the actual user code that does stuff) and one line (the emit) to make an asynchronous data exchange between threads. Sounds simple to me, but I can agree to disagree.

[alonhalawi]

I must have forgot the target, but as according to the examples I have, the placing of the function that I want to call as a thread is not done like this

QtCore.QThread(target= ***)
*

[kenchan]

Hey, why don't you point your questions to the Python group? Maybe some of those Python experts over there can help you.

[alonhalawi]

I'm considering moving the graphics to python. It's just exhausting missing some property, or widget attribute, or another feature that forces you to read a whole document of an object in qt ..

I do not go lengthly on the slot. I make a simple short change and go out, when it doesn't work I try looking for more examples that look similar to what I need.

I think I'll just do it with python, one line + one line + one line + noline with feature parameter queue-connection + reading another library kinda making me change my mind.

[alonhalawi]

Got it figured out, and done ...
with python.

Sorry Qt fans, I admit, I am a noob to Qt, I find it very non-scalable, however I truly recommend python (for graphics as well), and did it with less than a day (I studied most of the gui stuff)

simple, scalable, comfortable ... and most importantly, can adapt to more then one design option of mine.

[Chris Kawa]

If it's right for you then... well, it's right for you :)

Just as a comment - You made it in less then a day. Yup, for that size of apps Python might be the easier and faster option. But claiming Qt is not scalable... you clearly didn't do your research :)

[Chris Kawa]

Out of curiosity (and some boredom) I implemented a thread changing a color of a button every second. It took me < 5 minutes and there are exactly the 5 lines I was talking about needed to make threading work.
If you're not busy I would really like to see what it looks like in Python(without Qt). I don't mean to bash, I'm really curious.

@
class Worker : public QObject {
Q_OBJECT
public:
void doWork() {
while(true) {
QThread::currentThread()->sleep(1);
// 5 - emit the signal
emit foo(rand());
}
}
signals:
void foo(int);
};

class UI : public QPushButton {
public:
UI(QString text, QWidget* parent = nullptr) : QPushButton(text, parent) {
// 1 - start the thread
connect(this, &UI::clicked, &{ workerThread.start(); });
// 2 - fire up doWork in the thread
connect(&workerThread, &QThread::started, &worker, &Worker::doWork);
// 3 - modify ui in response to signal
connect(&worker, &Worker::foo, this, &UI::changeColor);
// 4 set thread affinity of worker object
worker.moveToThread(&workerThread);
}
private:
void changeColor(int color) {
setStyleSheet(QString("background-color: ") +
QColor::fromRgba(color).name());
}
Worker worker;
QThread workerThread;
};

int main(int argc, char *argv[])
{
QApplication a(argc, argv);
UI ui("hello");
ui.show();
return a.exec();
}@

[alonhalawi]

sure, I'll post it, and also, out of curiosity, I'll go over the code you've posted to see what I've missed.

Right now, I have to get the whole project done, and also I am going to a trip in Europe tonight, so I will post the code when I get back.

It will probably be next Saturday.

I didn't mean to be such an **hole in this post, but I really don't do gui, so ..

I guess I need to invest my time to learn Qt the way it should, before I judge the lang.