Why don't the signals emit from QThread ?

[daegontaven]

Hi, I'm using PyQt5 to build a python shell.
Can someone tell me why the signals emit late from QThread ?
Here is the StackOverflow question I posted.
Here is the gist in case someone needs to reference specific lines of code.

I've been at this for a long time. I can't seem to figure out how to make a FIFO delayed buffer for QTextEdit. Please, any help would be amazing. Thank you.

[SGaist]

Hi,

That code seems a bit unclean. You don't emit signals of another object.

Also and might be a silly question but: are you sure your code goes all the way to the signal emission ?
Are you sure that you have a slot connected on the other side ?

[daegontaven]

Yups, this is the line that connects the signal to the slot. Emitting it that way was the only I knew to get it working. BaseSignals() is just an intermediate class that helps me send signals between classes. Thanks for replying.

[SGaist]

Why not add the signals to your QThread based class ?

[daegontaven]

@SGaist You mean like so stream.py ? It still does not seem to work :(

[SGaist]

Would it be possible to reduce the problematic code to something simpler to test ?

[BjornW]

I don't really understand your code, but it is rarely a good idea to run Qt stuff inside a while(true) kind of construction.

[kshegunov]

I'm pretty sure your problem is your ConsoleStream object's in the main thread so it doesn't get the slots delivered. It's created here.
However, I share @SGaist's sentiment - you need to provide MREs if you expect people to take their time. No one wants to sift through dozens of classes to try to make sense of your code.

[daegontaven]

@BjornW @SGaist @kshegunov Okay I did what you guys wanted and added an MVCE so you guys can easily test the code. I hope this helps. Thank you so much for looking into this. I just saw you replies because I had given up on the project. But I feel now my efforts may not yet be in vain.

Here is the updated stackoverflow question. Here is a gist of main.ui and main.py so it's easier to highlight lines. Thank you once again.

^_^

[daegontaven]

@kshegunov said in Why don't the signals emit from QThread ?:

I'm pretty sure your problem is your ConsoleStream object's in the main thread so it doesn't get the slots delivered. It's created here.
However, I share @SGaist's sentiment - you need to provide MREs if you expect people to take their time. No one wants to sift through dozens of classes to try to make sense of your code.

I'll be referencing the gist from now on, so it's easier to locate where everything is.

Okay but PythonInterpreter is already inside it's own thread as shown here. Does it not apply to self.stream because of the multiple inheritance ? Or am I missing something ?

[BjornW]

Please provide a minimal example. There is too much, too confusing code :/

[daegontaven]

@BjornW What's confusing about the code I refactored ? There are no extra widgets, no unnecessary styling, no optimization of any sorts, lots of docstrings and just a few classes. Please point out what's confusing instead of saying it just is. Did you run the program ? It is literally two files and just 240 lines. I don't think it's easy to make a console for a Python interpreter under 160 lines using threads(Interpreter variables need to be accessed) . Ofcourse if it was that easy, I would have solved it through looking at documentation. This is the absolute minimal version. I only used parts detrimental to the functionality of a console.

Sorry. My temper got the best of me.. I've been at this too long and no one seems to know the answer. The simplest version which was 160 lines didn't even have a buffer and couldn't run a while loop. I keep telling people it's not possible. They keep telling me MCVE. Can't take the engine out and call it a car. Can't the take the chassis out and try to drive it either. Tell me what I can do to make this better instead of just saying MCVE. What do I MCVE.. Is there even anything left ?

I really need some coffee.

[BjornW]

@daegontaven

Its cool. I'm not saying your code is bad, I'm just saying it is not very easy to jump into someone elses code, regardless how good it is.

Now, if you create a thread, and by that i mean just a thread. No buffers and stuff, and create some object to emit a signal, is it "late"? I suppose not, after that, just keep adding stuff until you can actually reproduce the "late" behavior with the absolutely least amount of code. It is a good idea, even for your own sake, to try to resolve the issue like that. It is very easy to get lost in other workings of your code.

I'd like to try the following:

In your DelayedBuffer you create a timer like so:

L50: self.timer = QTimer()

This timer is not a child of the DelayedBuffer and therefore will not move with the buffer into a new thread. This means it will be executing in the thread where it was created (?). Try changing to

L50: self.timer = QTimer(self)

Otherwise:
Have you tried flushing your outputs?
Why do you call processEvents?

[kshegunov]

@daegontaven said in Why don't the signals emit from QThread ?:

Okay but PythonInterpreter is already inside it's own thread as shown here. Does it not apply to self.stream because of the multiple inheritance ? Or am I missing something ?

That a QObject belongs to a thread is something Qt introduces. In regular C++ there's no such thing - the data (object) is separate from the thread the methods are run in. In your case the constructor of PythonInterpreter is run in the same thread as the one the calling function is run in, meaning everything you do in that constructor is again in the same thread. As you're creating an object in said constructor that object is going to be in the same thread, understand where I'm getting with this? If you later call moveToThread that applies only to the one object, not to all the objects created in the constructor. So to make this work properly Qt has introduced object ownership - i.e. one object is a child to another object. If an object's moved to a thread all its children are moved to the same thread. However you don't set a parent to the object you create in the constructor so those objects have no parent, thus are not children to the PythonInterpreter instance, and finally are not going to be moved to the PythonInterpreter's object's thread. There are 2 things that can be done to solve this:

1. Always set the parent for any QObject instance you create, so you have a nice tidy and predictable object tree.
2. Move all the objects manually to the required thread, which is quite cumbersome.

More on ownership and trees you can find here and QObject::moveToThread documentation specifically refenreces which objects are moved and what are the limitations of this function.

[daegontaven]

@kshegunov I updated the gist. Is this how you set a parent ?

Or do I have to call setParent ? Which object do I call that method on ?

[kshegunov]

self.stream = ConsoleStream(this)

would be my guess (I don't know python, so you need to adjust that line accordingly).

Which object do I call that method on ?

For each QObject derived class' instance you want to be moved with the parent to the thread. As is now, the PythonInterpreter instance is moved to a thread, here:

self.interpreter.moveToThread(self.thread)

but the objects you create in its constructor, i.e:

self.stream = ConsoleStream()

are not, as they have no parent associated with them.

[daegontaven]

@kshegunov Thank you so much for responding. I tried your method(setting a parent). It actually improved the response time significantly. Much less lag on windows machines now. But the blocking issue still persists. It would appear this issue has never been solved in all these years. So I've decided to abandon the QPlainTextEdit strategy. Is it possible to use QListView instead? How would I go about changing the code so it would work as close as possible to QPlainTextEdit?

[SGaist]

It's not a question of unsolved issue, QPlainTextEdit has never been designed to handle console like behaviour. AFAIK, what people do is to use a heavily styled QListView + QStyleItemDelegate combo with a custom model to allow more fine grained updates and avoid performance issue by using a rolling window over the data.

[daegontaven]

@SGaist Yes, that's what I meant to say. I wish it was built for faster processing of live data too. I think the issue with the signal from the buffer was that the emits are being queued in the event loop and unfortunately Python has a GIL which might be blocking the concurrent execution. So it might be an issue inherent to Python and not Qt per se.

As for the QListView, I think I understand what you're getting at when you say"rolling window". I did initially use QListView before ever starting with a buffer, but I saw that the performance was still bad. But now I think it was because like there is a setMaximumBlockCount() method for QPlainTextEdit, I never removed the items from the model. I was using QStringListModel and not a custom model. However, I am not familiar QStyledItemDelegate or custom models either. Could you direct me to some sources or examples that implement something like this? Thank you so much for the help.

Edit:
After some digging around, I found promising documentation for model view programming. I think this is what I was looking for. I want to thank everyone for sticking with my mess and helping me anyway. I'll mark this thread as closed. Thank you guys/gals once again.

[daegontaven]

Guys/Gals, I guess I gave up too quickly because someone on stack-overflow solved the problem. I'm just going to leave the solution here in case someone needs it in the future.

Here is the change that fixed the issue.