Show QProgressbar with computationally heavy background process

[janhein_dejong]

I'm building an application that let's the user export his/her work. This is a computationally heavy process, lasting for a minute or so, during which I want to show a progress bar (and make the rest of the UI unresponsive).

I've tried the implementation below, which works fine for a non-computationally expensive background process (e.g. waiting for 0.1 s). However, for a CPU heavy process, the UI becomes very laggy and unresponsive (but not completely unresponsive).

Any idea how I can solve this?

import sys
import time

from PySide2 import QtCore
from PySide2.QtCore import Qt
import PySide2.QtWidgets as QtWidgets


class MainWindow(QtWidgets.QMainWindow):
    """Main window, with one button for exporting stuff"""

    def __init__(self, parent=None):
        super().__init__(parent)
        central_widget = QtWidgets.QWidget(self)
        layout = QtWidgets.QHBoxLayout(self)
        button = QtWidgets.QPushButton("Press me...")
        button.clicked.connect(self.export_stuff)
        layout.addWidget(button)
        central_widget.setLayout(layout)
        self.setCentralWidget(central_widget)

    def export_stuff(self):
        """Opens dialog and starts exporting"""
        some_window = MyExportDialog(self)
        some_window.exec_()


class MyAbstractExportThread(QtCore.QThread):
    """Base export thread"""
    change_value = QtCore.Signal(int)

    def run(self):
        cnt = 0
        while cnt < 100:
            cnt += 1
            self.operation()
            self.change_value.emit(cnt)

    def operation(self):
        pass


class MyExpensiveExportThread(MyAbstractExportThread):

    def operation(self):
        """Something that takes a lot of CPU power"""
        some_val = 0
        for i in range(1000000):
            some_val += 1


class MyInexpensiveExportThread(MyAbstractExportThread):

    def operation(self):
        """Something that doesn't take a lot of CPU power"""
        time.sleep(.1)


class MyExportDialog(QtWidgets.QDialog):
    """Dialog which does some stuff, and shows its progress"""

    def __init__(self, parent=None):
        super().__init__(parent, Qt.WindowCloseButtonHint)
        self.setWindowTitle("Exporting...")
        layout = QtWidgets.QHBoxLayout()
        self.progress_bar = self._create_progress_bar()
        layout.addWidget(self.progress_bar)
        self.setLayout(layout)
        self.worker = MyInexpensiveExportThread()  # Works fine
        # self.worker = MyExpensiveExportThread()  # Super laggy
        self.worker.change_value.connect(self.progress_bar.setValue)
        self.worker.start()
        self.worker.finished.connect(self.close)

    def _create_progress_bar(self):
        progress_bar = QtWidgets.QProgressBar(self)
        progress_bar.setMinimum(0)
        progress_bar.setMaximum(100)
        return progress_bar


if __name__ == "__main__":
    app = QtWidgets.QApplication()
    window = MainWindow()
    window.show()
    sys.exit(app.exec_())

[janhein_dejong]

Ok... it took a bit of time, but I finally figured this one out.

The difficulty with showing a responsive user-interface while running a computationally heavy process using threading, stems from the fact in this case one combines a so-called IO-bound thread (i.e. the GUI), with a CPU-bound thread (i.e. the computation). For a IO-bound process, the time it takes to complete is defined by the fact that the thread has to wait on input or output (e.g. a user clicking on things, or a timer). By contrast, the time required to finish a CPU-bound process is limited by the power of the processing unit performing the process.

In principle, mixing these types of threads in Python should not be a problem. Although the GIL enforces that only one thread is running at a single instance, the operating system in fact splits the processes up into smaller instructions, and switches between them. If a thread is running, it has the GIL and executes some of its instructions. After a fixed amount of time, it needs to release the GIL. Once released, the GIL can schedule activate any other 'runnable' thread - including the one that was just released.

The problem however, is with the scheduling of these threads. Here things become a bit fuzzy for me, but basically what happens is that the CPU-bound thread seems to dominate this selection, from what I could gather due to a process called the "convey effect". Hence, the erratic and unpredictable behavior of a Qt GUI when running a CPU-bound thread in the background.

I found some interesting reading material on this:

• Understanding the GIL
• More in depth analysis of the GIL
• Nice visual representation of thread scheduling

So... this is very nice and all, how do we fix this?

In the end, I managed to get what I want using multiprocessing. This allows you to actually run a process parallel to the GUI, instead in sequential fashion. This ensures the GUI stays as responsive as it would be without the CPU-bound process in the background.

Multiprocessing has a lot of difficulties of its own, for example the fact that sending information back and forth between processes is done by sending pickled objects across a pipeline. However, the end-result is really superior in my case.

Below I put a code snippet, showing my solution. It contains a class called ProgressDialog, which provides an easy API for setting this up with your own CPU-bound process.

"""Contains class for executing a long running process (LRP) in a separate
process, while showing a progress bar"""

import multiprocessing as mp

from PySide2 import QtCore
from PySide2.QtCore import Qt
import PySide2.QtWidgets as QtWidgets


class ProgressDialog(QtWidgets.QDialog):
    """Dialog which performs a operation in a separate process, shows a
    progress bar, and returns the result of the operation

    Parameters
    ----
    title: str
        Title of the dialog
    operation: callable
        Function of the form f(conn, *args) that will be run
    args: tuple
        Additional arguments for operation
    parent: QWidget
        Parent widget

    Returns
    ----
    result: int
        The result is an integer. A 0 represents successful completion, or
        cancellation by the user. Negative numbers represent errors. -999
        is reserved for any unforeseen uncaught error in the operation.

    Examples
    ----
    The function passed as the operation parameter should be of the form
    ``f(conn, *args)``. The conn argument is a Connection object, used to
    communicate the progress of the operation to the GUI process. The
    operation can pass its progress with a number between 0 and 100, using
    ``conn.send(i)``. Once the process is finished, it should send 101.
    Error handling is done by passing negative numbers.

    >>> def some_function(conn, *args):
    >>>     conn.send(0)
    >>>     a = 0
    >>>     try:
    >>>         for i in range(100):
    >>>                 a += 1
    >>>                 conn.send(i + 1)  # Send progress
    >>>     except Exception:
    >>>         conn.send(-1)  # Send error code
    >>>     else:
    >>>         conn.send(101)  # Send successful completion code

    Now we can use an instance of the ProgressDialog class within any 
    QtWidget to execute the operation in a separate process, show a progress 
    bar, and print the error code:

    >>> progress_dialog = ProgressDialog("Running...", some_function, self)
    >>> progress_dialog.finished.connect(lambda err_code: print(err_code))
    >>> progress_dialog.open()
    """

    def __init__(self, title, operation, args=(), parent=None):
        super().__init__(parent, Qt.WindowCloseButtonHint)
        self.setWindowTitle(title)
        self.progress_bar = QtWidgets.QProgressBar(self)
        self.progress_bar.setValue(0)
        layout = QtWidgets.QHBoxLayout()
        layout.addWidget(self.progress_bar)
        self.setLayout(layout)

        # Create connection pipeline
        self.parent_conn, self.child_conn = mp.Pipe()

        # Create process
        args = (self.child_conn, *args)
        self.process = mp.Process(target=operation, args=args)

        # Create status emitter
        self.progress_emitter = ProgressEmitter(self.parent_conn, self.process)
        self.progress_emitter.signals.progress.connect(self.slot_update_progress)
        self.thread_pool = QtCore.QThreadPool()

    def slot_update_progress(self, i):
        if i < 0:
            self.done(i)
        elif i == 101:
            self.done(0)
        else:
            self.progress_bar.setValue(i)

    def open(self):
        super().open()
        self.process.start()
        self.thread_pool.start(self.progress_emitter)

    def closeEvent(self, *args):
        self.progress_emitter.running = False
        self.process.terminate()
        super().closeEvent(*args)


class ProgressEmitter(QtCore.QRunnable):
    """Listens to status of process"""

    class ProgressSignals(QtCore.QObject):
        progress = QtCore.Signal(int)

    def __init__(self, conn, process):
        super().__init__()
        self.conn = conn
        self.process = process
        self.signals = ProgressEmitter.ProgressSignals()
        self.running = True

    def run(self):
        while self.running:
            if self.conn.poll():
                progress = self.conn.recv()
                self.signals.progress.emit(progress)
                if progress < 0 or progress == 101:
                    self.running = False
            elif not self.process.is_alive():
                self.signals.progress.emit(-999)
                self.running = False

[JonB]

@janhein_dejong
First, your real-life case may not be as bad as your tight-loop example.

Second, my machines are often UI-laggy if you do have a really heavy computation going on, doesn't matter whether it's separate processes or threads, c'est la vie!

Third, it gets complicated, but because you're using Python apparently multi-threading works such that only one thread at a time can use the Python interpreter. I don't know how this plays with a PyQt program, but it could mean your threads are effectively serialized.

And fourth & finally, again I don't know how that plays with Qt/PyQt, but I have read that from Python you are really supposed to use Python threading, not Qt threading.

See if you get a more sympathetic answer! You could Google a bit for Python/PyQt threading. If you don't get a better answer here, you could join the PyQt mailing list and ask there for any PyQt-specifics.

[Denni 0]

Okay first and foremost -- do not sub-class QThread as that is Python-Qt4 not Python-Qt5 and while you may not notice it using it has issues that is why it was implemented differently in the newer version.

Next as @JonB states when using Python regardless of what threading you are using you have to deal with the GIL which basically runs Threads in a serial manner with only one Thread having control of the process at a time. This means if you have a GUI that you want to be responsive and you start up a intensively long running process you need to build into that LRP breaks that hand back control to the Event Handler periodically or everything else just seems like it becomes unresponsive as the Event Handler is not getting called. Note I say seems because the events are occurring and stacking up in the Event Queue they are just not being handled.

Next @JonB is incorrect in his statement about which Threading module you should use. If you are using Python-Qt you should use QThread as it works with the QApplication Event Handler better (aka the other will have issues) but if you are not using Python-Qt then yeah you should use the python threading.

Lastly I have extensive knowledge/experience with Python-Qt QThreading and can help you out here or in my free online classroom as well. Just ask questions

[JonB]

@Denni-0 said in Show QProgressbar with computationally heavy background process:

Next @JonB is incorrect in his statement about which Threading module you should use. If you are using Python-Qt you should use QThread as it works with the QApplication Event Handler better

Fair enough, you have more experience than I in this area.

@janhein_dejong
https://www.learnpyqt.com/courses/concurrent-execution/multithreading-pyqt-applications-qthreadpool/ looks up-to-date and is an interesting read. I think that is recommending QRunnable and the QThreadPool. There is also a discussion in https://stackoverflow.com/questions/6783194/background-thread-with-qthread-in-pyqt.

[janhein_dejong]

Thanks so much, I'll go ahead and implement these suggestions. @JonB what do you mean by LRP breaks?

[JonB]

@janhein_dejong said in Show QProgressbar with computationally heavy background process:

@JonB what do you mean by LRP breaks?

It was @Denni-0 who wrote "LRP breaks", not I. It will stand for "Long Running Processes". I think he means you will want to call https://doc.qt.io/qt-5/qcoreapplication.html#processEvents (which has its issues) periodically during a "LRP" in order to give the UI time to do stuff in order to keep it responsive.

[janhein_dejong]

Alright... I turned it into the example below. Already works much better... the tight loop is still somewhat laggy, but at least it reaches 100% when you expect it to. I'll try it on my real world example, to see how that behaves.

import sys
import time

from PySide2 import QtCore
from PySide2.QtCore import Qt
import PySide2.QtWidgets as QtWidgets


class MainWindow(QtWidgets.QMainWindow):
    """Main window, with one button for exporting stuff"""

    def __init__(self, parent=None):
        super().__init__(parent)
        central_widget = QtWidgets.QWidget(self)
        layout = QtWidgets.QHBoxLayout(self)
        button1 = QtWidgets.QPushButton("Tight operation")
        button2 = QtWidgets.QPushButton("Loose operation")
        button1.clicked.connect(self.export_tight_stuff)
        button2.clicked.connect(self.export_loose_stuff)
        layout.addWidget(button1)
        layout.addWidget(button2)
        central_widget.setLayout(layout)
        self.setCentralWidget(central_widget)

    def export_tight_stuff(self):
        """Opens dialog and starts exporting"""
        worker = Worker(self.tight_operation)
        some_window = MyExportDialog(worker, self)
        some_window.exec_()

    def export_loose_stuff(self):
        """Opens dialog and starts exporting"""
        worker = Worker(self.loose_operation)
        some_window = MyExportDialog(worker, self)
        some_window.exec_()

    @staticmethod
    def loose_operation():
        """Something that doesn't take a lot of CPU power"""
        time.sleep(.1)

    @staticmethod
    def tight_operation():
        """Something that takes a lot of CPU power"""
        some_val = 0
        for i in range(1_000_000):
            some_val += 1


class WorkerSignals(QtCore.QObject):
    progress = QtCore.Signal(int)
    finished = QtCore.Signal()


class Worker(QtCore.QRunnable):

    def __init__(self, fn):
        super().__init__()
        self.operation = fn
        self.signals = WorkerSignals()

    def run(self):
        cnt = 0
        while cnt < 100:
            cnt += 1
            self.operation()
            self.signals.progress.emit(cnt)
        self.signals.finished.emit()


class MyExportDialog(QtWidgets.QDialog):
    """Dialog which does some stuff, and shows it's progress"""

    def __init__(self, worker, parent=None):
        super().__init__(parent, Qt.WindowCloseButtonHint)
        self.setWindowTitle("Exporting...")
        layout = QtWidgets.QHBoxLayout()
        self.progress_bar = QtWidgets.QProgressBar(self)
        layout.addWidget(self.progress_bar)
        self.setLayout(layout)
        worker.signals.progress.connect(self.progress_bar.setValue)
        worker.signals.finished.connect(self.close)
        self.thread_pool = QtCore.QThreadPool()
        self.thread_pool.start(worker)


if __name__ == "__main__":
    app = QtWidgets.QApplication()
    window = MainWindow()
    window.show()
    sys.exit(app.exec_())

[Denni 0]

@janhein_dejong are you aware of the issues that using super( ) introduces to your programs or are you even aware of the reason it was even created? I ask this because so many folks use this without knowing the potential damage they are creating while not using super( ) does not introduce new issues and has only one rather rare case where using super( ) was meant to fix

Basically let us put it this way not using super( ) has 1 rare issue but using super( ) introduces at least 3 new much more common potential issues. I think the math right there would deter the use of super( ) but for some reason some still promote jumping out of a plane without a parachute and convincingly enough that many folks seem willing to follow them.

Next if you are using pyqt5 which I believe is the case then sys.exit(app.exec_()) should be replaced with app.exec() as the previous is PyQt4 and the latter is PyQt5

Also QThreadPool is I believe sub-classable but then what it is used for is creating a container for reusable threads -- aka use it for some task then use it again for another task over and over again -- you would not use this for CRP (Continuous Running Process) but if its a self contained LRP (Long Running Process) that just does something and then you might need that thread for something else once its done then the overhead for QThreadPool might be worth using it. It is all about trade offs here.

In your example code the tight_operation is a Thread blocker and once it starts nothing else will be able to do anything within the same process until it finishes -- most folks do not take into account that in Python Threads are not separate processes they are simply a different thread within the same process -- its kind of like having a hose that has water running through it and 3 folks that need to use, A just wants a drink, B wants to wash their car and C wants to fill their large pool and if the CPU happens to hand the hose to C that will mean A and B will need to wait until C is done before they can gain access to the hose at all. This is basically how the GIL works within Python. However if C were to hand the hose back to CPU then CPU would go to A or B (depending on who was next in queue) and let them have the hose.

So if you understand how the GIL works you need to make a computensive function into smaller sequential units and then between each unit you hand control back to the Event Handler.

On the other hand if you are needing to run a computensive task that you cannot break into smaller pieces in order to pass the control to the Event handler then you will need to use multiprocessing.

[janhein_dejong]

Hmm... good questions @Denni-0

Regarding super, do you mean it's better to use the syntax below? What are these dangers you speak of?

class SubClass: 

    def __init__(self, *args):
        SuperClass.__init__(self, *args)

Regarding sys.exit(app.exec_()) - the official Qt for Python page shows an example using that method. I'm using PySide2 (Qt for Python), so I guess for that binding that's still the way to go.

Regarding subclassing QThreadPool - I'm not sure I get your point completely. Should I subclass it? What's the advantage?

Regarding threading in Python - you're right I'm not entirely sure how this works internally. My understanding was that if you have two threads (A and B), the GIL will try to break these threads into smaller operations, and execute these chunks from both threads on the same core in your CPU sequentially. So in my example, the GIL breaks tight_operation into smaller pieces, and switches between the GUI thread, and the QThreadPool initiated thread between these smaller pieces; not solely upon completion of tight_operation. The GUI thread should therefore not be blocked by tight_operation. Am I right?

[Denni 0]

No that is not what I meant - what I mean is unless you fully understand how to use super( ) do not use it at all period (see example below) as there are at least three things you must be aware of (which I do not have memorized) when using Super( ) to make sure you use it correctly. Lastly because the situation for actually using Super( ) is rather rare the likelihood of needing to use it will be rare to non-existent thus you will most likely not run into that single rare case with most of the programs you write.

class SubClass(Object):
    def __init__(self):
        Object.__init__(self)

As for the "Official Qt for Python" documentation showing PyQt4 examples -- yes they have issues with some of their documentation not being current. For instance QThread which should not be sub-classed in PyQt5 still has documentation showing it being sub-classed because that was how they were handling it in PyQt4 which is something I got burned on. Also I only learned recently about this sys.exit thing and had been using it prior to when I learned about it. Thus Qt being behind in updating their documentation is not really anything new to me, which is frustrating at times since I kind of depend on the documentation too. Still I shared the information with you. Now what you choose to do with that information is completely up to you. As for me when I learned about it I changed how I was using it and I share that with others who have not heard about it yet.

Next the advantage to sub-classing QThreadPool is the same advantage you get when subclassing any Object all I just said was I believed it could be sub-classed as opposed to QThread which should not be sub-classed. In case you wanted to do so.

No you are wrong as Python does not magically break your code up into smaller chunks -- I mean how would it know where to break it, code only takes a break when you program it to do so -- thus Python just runs it the way you have it coded. Again all it does is give the CPU handle to a process/function and then waits for that process/function to give that handle back and it will wait indefinitely for that handle. This is why long running loops can freeze up your Gui even though it has been placed into its own Thread be it a QThread or a QThreadPool -- the GIL simply states I will run each process one at a time sequentially finishing the first process before moving on to the next. So how can my code benefit from this you might ask ... well basically Gui's are notorious for long periods of inactivity and if the threaded processes are short enough the amount of time it takes for them to process is unnoticed by the user -- however those kind of processes do not really benefit from Threading so there really is no sense in adding that overhead to handle them. That is unless you have like a series of processes that need to run sequentially while other things are taking place in the Gui but then you have just broken a large process down into smaller chunks and I would assume you would hand control back to Event Handler between each of these processes. Now the cash cow for Threads are the long running processes or the continuous running process as these benefit from Threads greatly BUT ONLY IF the coder is smart about the process and understands the restrictions of the GIL and they code their long/continuous running process such that it executes smaller chunks and gives back control to the Event Handler periodically such as in the following:

while not EoF:
    ReadFromFile()
    DoSomething()
    QCoreApplication.processEvents()

So the above is a long running function -- and if we assume the file has 10 of 1000s of lines (aka at least 10,000 records or more) that need to process and we are processing a Record (or we could set it up to process a few) and then we give control back to the Event Handler before we run the next Record (or series of Records) -- this kind of function can thus utilize a QThread efficiently but care must be taken when using QCoreApplication.processEvents() as Qt states that using it can be problematic and that is about all I have seen them say about -- no details on how it can be problematic -- although it might be out there somewhere. Still so far I have not encountered any issues with it but then I think I might understand how this works and I believe have been using it appropriately (but I could be wrong as again I have not heard exactly how this thing actually works or what its potential issues might be -- I have simply been going off of past knowledge on similar functionality that exists within other Event Handling languages that I have used but I am continuously aware that it can have issues so I watch for them).

I hope that answers all your questions but please if they raised any more questions just ask -- to me the only bad question is the unasked question.

[janhein_dejong]

Ok... it took a bit of time, but I finally figured this one out.

The difficulty with showing a responsive user-interface while running a computationally heavy process using threading, stems from the fact in this case one combines a so-called IO-bound thread (i.e. the GUI), with a CPU-bound thread (i.e. the computation). For a IO-bound process, the time it takes to complete is defined by the fact that the thread has to wait on input or output (e.g. a user clicking on things, or a timer). By contrast, the time required to finish a CPU-bound process is limited by the power of the processing unit performing the process.

In principle, mixing these types of threads in Python should not be a problem. Although the GIL enforces that only one thread is running at a single instance, the operating system in fact splits the processes up into smaller instructions, and switches between them. If a thread is running, it has the GIL and executes some of its instructions. After a fixed amount of time, it needs to release the GIL. Once released, the GIL can schedule activate any other 'runnable' thread - including the one that was just released.

The problem however, is with the scheduling of these threads. Here things become a bit fuzzy for me, but basically what happens is that the CPU-bound thread seems to dominate this selection, from what I could gather due to a process called the "convey effect". Hence, the erratic and unpredictable behavior of a Qt GUI when running a CPU-bound thread in the background.

I found some interesting reading material on this:

• Understanding the GIL
• More in depth analysis of the GIL
• Nice visual representation of thread scheduling

So... this is very nice and all, how do we fix this?

In the end, I managed to get what I want using multiprocessing. This allows you to actually run a process parallel to the GUI, instead in sequential fashion. This ensures the GUI stays as responsive as it would be without the CPU-bound process in the background.

Multiprocessing has a lot of difficulties of its own, for example the fact that sending information back and forth between processes is done by sending pickled objects across a pipeline. However, the end-result is really superior in my case.

Below I put a code snippet, showing my solution. It contains a class called ProgressDialog, which provides an easy API for setting this up with your own CPU-bound process.

"""Contains class for executing a long running process (LRP) in a separate
process, while showing a progress bar"""

import multiprocessing as mp

from PySide2 import QtCore
from PySide2.QtCore import Qt
import PySide2.QtWidgets as QtWidgets


class ProgressDialog(QtWidgets.QDialog):
    """Dialog which performs a operation in a separate process, shows a
    progress bar, and returns the result of the operation

    Parameters
    ----
    title: str
        Title of the dialog
    operation: callable
        Function of the form f(conn, *args) that will be run
    args: tuple
        Additional arguments for operation
    parent: QWidget
        Parent widget

    Returns
    ----
    result: int
        The result is an integer. A 0 represents successful completion, or
        cancellation by the user. Negative numbers represent errors. -999
        is reserved for any unforeseen uncaught error in the operation.

    Examples
    ----
    The function passed as the operation parameter should be of the form
    ``f(conn, *args)``. The conn argument is a Connection object, used to
    communicate the progress of the operation to the GUI process. The
    operation can pass its progress with a number between 0 and 100, using
    ``conn.send(i)``. Once the process is finished, it should send 101.
    Error handling is done by passing negative numbers.

    >>> def some_function(conn, *args):
    >>>     conn.send(0)
    >>>     a = 0
    >>>     try:
    >>>         for i in range(100):
    >>>                 a += 1
    >>>                 conn.send(i + 1)  # Send progress
    >>>     except Exception:
    >>>         conn.send(-1)  # Send error code
    >>>     else:
    >>>         conn.send(101)  # Send successful completion code

    Now we can use an instance of the ProgressDialog class within any 
    QtWidget to execute the operation in a separate process, show a progress 
    bar, and print the error code:

    >>> progress_dialog = ProgressDialog("Running...", some_function, self)
    >>> progress_dialog.finished.connect(lambda err_code: print(err_code))
    >>> progress_dialog.open()
    """

    def __init__(self, title, operation, args=(), parent=None):
        super().__init__(parent, Qt.WindowCloseButtonHint)
        self.setWindowTitle(title)
        self.progress_bar = QtWidgets.QProgressBar(self)
        self.progress_bar.setValue(0)
        layout = QtWidgets.QHBoxLayout()
        layout.addWidget(self.progress_bar)
        self.setLayout(layout)

        # Create connection pipeline
        self.parent_conn, self.child_conn = mp.Pipe()

        # Create process
        args = (self.child_conn, *args)
        self.process = mp.Process(target=operation, args=args)

        # Create status emitter
        self.progress_emitter = ProgressEmitter(self.parent_conn, self.process)
        self.progress_emitter.signals.progress.connect(self.slot_update_progress)
        self.thread_pool = QtCore.QThreadPool()

    def slot_update_progress(self, i):
        if i < 0:
            self.done(i)
        elif i == 101:
            self.done(0)
        else:
            self.progress_bar.setValue(i)

    def open(self):
        super().open()
        self.process.start()
        self.thread_pool.start(self.progress_emitter)

    def closeEvent(self, *args):
        self.progress_emitter.running = False
        self.process.terminate()
        super().closeEvent(*args)


class ProgressEmitter(QtCore.QRunnable):
    """Listens to status of process"""

    class ProgressSignals(QtCore.QObject):
        progress = QtCore.Signal(int)

    def __init__(self, conn, process):
        super().__init__()
        self.conn = conn
        self.process = process
        self.signals = ProgressEmitter.ProgressSignals()
        self.running = True

    def run(self):
        while self.running:
            if self.conn.poll():
                progress = self.conn.recv()
                self.signals.progress.emit(progress)
                if progress < 0 or progress == 101:
                    self.running = False
            elif not self.process.is_alive():
                self.signals.progress.emit(-999)
                self.running = False

[Denni 0]

@janhein_dejong nice in depth perusal of the inner workings nice to know this but for most beginners it is not going to worth a whole lot ;-) just too much information --- the basic principles I believe I pointed out -- and yeah you can use a multiprocessing to do this or you could have learned how to handle Event driven processing with Threads properly as well. I am not saying multiprocessing is the wrong solution -- but you are adding an additional layer of complexity that I am not sure was even necessary. K.I.S.S. (Keep It Simple and Smart) is always the best way to go in any program and I think a simpler solution just might have worked.

Also keep in mind an issue with Multiprocessing is that the Sender sending data down a pipe must notify the Receiver that data has been sent -- as far as pickling is concerned in most (but not all) situations where you might need this the data you are sending is a non-issue -- you just have to make sure you do not send anything that cannot be pickled

Still in order to set up the communication Pipe effectively you need to create a listening thread on the Receiving end that responds to the -- data sent message (which by the way is not a Signal because Signals/Slots do not work across Process boundaries) so it can pull the sent data from the pipe and then signals its parent that data has been received so you are still going to be dealing with a Thread and needing to be sure you are handling the Event stuff correctly

Again thanks for that extra information -- I hope this helps you understand what you just stepped into ;-)

@janhein_dejong said in Show QProgressbar with computationally heavy background process:

In principle, mixing these types of threads in Python should not be a problem. Although the GIL enforces that only one thread is running at a single instance, the operating system in fact splits the processes up into smaller instructions, and switches between them. If a thread is running, it has the GIL and executes some of its instructions. After a fixed amount of time, it needs to release the GIL. Once released, the GIL can schedule activate any other 'runnable' thread - including the one that was just released.

I want to say to this that "In principle" is kind of like saying "In Theory" -- and the take-away from this is while this is kind how it is supposed to work in actuality it does not work this way -- in short if you set up an endless while loop within a single process I can pretty much guarantee that your GUI is going to freeze up as that is the reality of what happens. Thus as I said you must learn how to handle Event driven code in a proper manner so that these secondary tasks remain secondary tasks and do not end up taking over the CPU rendering the rest of your code inert