Show QProgressbar with computationally heavy background process

[janhein_dejong]

I'm building an application that let's the user export his/her work. This is a computationally heavy process, lasting for a minute or so, during which I want to show a progress bar (and make the rest of the UI unresponsive).

I've tried the implementation below, which works fine for a non-computationally expensive background process (e.g. waiting for 0.1 s). However, for a CPU heavy process, the UI becomes very laggy and unresponsive (but not completely unresponsive).

Any idea how I can solve this?

import sys
import time

from PySide2 import QtCore
from PySide2.QtCore import Qt
import PySide2.QtWidgets as QtWidgets


class MainWindow(QtWidgets.QMainWindow):
    """Main window, with one button for exporting stuff"""

    def __init__(self, parent=None):
        super().__init__(parent)
        central_widget = QtWidgets.QWidget(self)
        layout = QtWidgets.QHBoxLayout(self)
        button = QtWidgets.QPushButton("Press me...")
        button.clicked.connect(self.export_stuff)
        layout.addWidget(button)
        central_widget.setLayout(layout)
        self.setCentralWidget(central_widget)

    def export_stuff(self):
        """Opens dialog and starts exporting"""
        some_window = MyExportDialog(self)
        some_window.exec_()


class MyAbstractExportThread(QtCore.QThread):
    """Base export thread"""
    change_value = QtCore.Signal(int)

    def run(self):
        cnt = 0
        while cnt < 100:
            cnt += 1
            self.operation()
            self.change_value.emit(cnt)

    def operation(self):
        pass


class MyExpensiveExportThread(MyAbstractExportThread):

    def operation(self):
        """Something that takes a lot of CPU power"""
        some_val = 0
        for i in range(1000000):
            some_val += 1


class MyInexpensiveExportThread(MyAbstractExportThread):

    def operation(self):
        """Something that doesn't take a lot of CPU power"""
        time.sleep(.1)


class MyExportDialog(QtWidgets.QDialog):
    """Dialog which does some stuff, and shows its progress"""

    def __init__(self, parent=None):
        super().__init__(parent, Qt.WindowCloseButtonHint)
        self.setWindowTitle("Exporting...")
        layout = QtWidgets.QHBoxLayout()
        self.progress_bar = self._create_progress_bar()
        layout.addWidget(self.progress_bar)
        self.setLayout(layout)
        self.worker = MyInexpensiveExportThread()  # Works fine
        # self.worker = MyExpensiveExportThread()  # Super laggy
        self.worker.change_value.connect(self.progress_bar.setValue)
        self.worker.start()
        self.worker.finished.connect(self.close)

    def _create_progress_bar(self):
        progress_bar = QtWidgets.QProgressBar(self)
        progress_bar.setMinimum(0)
        progress_bar.setMaximum(100)
        return progress_bar


if __name__ == "__main__":
    app = QtWidgets.QApplication()
    window = MainWindow()
    window.show()
    sys.exit(app.exec_())

[janhein_dejong]

Ok... it took a bit of time, but I finally figured this one out.

The difficulty with showing a responsive user-interface while running a computationally heavy process using threading, stems from the fact in this case one combines a so-called IO-bound thread (i.e. the GUI), with a CPU-bound thread (i.e. the computation). For a IO-bound process, the time it takes to complete is defined by the fact that the thread has to wait on input or output (e.g. a user clicking on things, or a timer). By contrast, the time required to finish a CPU-bound process is limited by the power of the processing unit performing the process.

In principle, mixing these types of threads in Python should not be a problem. Although the GIL enforces that only one thread is running at a single instance, the operating system in fact splits the processes up into smaller instructions, and switches between them. If a thread is running, it has the GIL and executes some of its instructions. After a fixed amount of time, it needs to release the GIL. Once released, the GIL can schedule activate any other 'runnable' thread - including the one that was just released.

The problem however, is with the scheduling of these threads. Here things become a bit fuzzy for me, but basically what happens is that the CPU-bound thread seems to dominate this selection, from what I could gather due to a process called the "convey effect". Hence, the erratic and unpredictable behavior of a Qt GUI when running a CPU-bound thread in the background.

I found some interesting reading material on this:

• Understanding the GIL
• More in depth analysis of the GIL
• Nice visual representation of thread scheduling

So... this is very nice and all, how do we fix this?

In the end, I managed to get what I want using multiprocessing. This allows you to actually run a process parallel to the GUI, instead in sequential fashion. This ensures the GUI stays as responsive as it would be without the CPU-bound process in the background.

Multiprocessing has a lot of difficulties of its own, for example the fact that sending information back and forth between processes is done by sending pickled objects across a pipeline. However, the end-result is really superior in my case.

Below I put a code snippet, showing my solution. It contains a class called ProgressDialog, which provides an easy API for setting this up with your own CPU-bound process.

"""Contains class for executing a long running process (LRP) in a separate
process, while showing a progress bar"""

import multiprocessing as mp

from PySide2 import QtCore
from PySide2.QtCore import Qt
import PySide2.QtWidgets as QtWidgets


class ProgressDialog(QtWidgets.QDialog):
    """Dialog which performs a operation in a separate process, shows a
    progress bar, and returns the result of the operation

    Parameters
    ----
    title: str
        Title of the dialog
    operation: callable
        Function of the form f(conn, *args) that will be run
    args: tuple
        Additional arguments for operation
    parent: QWidget
        Parent widget

    Returns
    ----
    result: int
        The result is an integer. A 0 represents successful completion, or
        cancellation by the user. Negative numbers represent errors. -999
        is reserved for any unforeseen uncaught error in the operation.

    Examples
    ----
    The function passed as the operation parameter should be of the form
    ``f(conn, *args)``. The conn argument is a Connection object, used to
    communicate the progress of the operation to the GUI process. The
    operation can pass its progress with a number between 0 and 100, using
    ``conn.send(i)``. Once the process is finished, it should send 101.
    Error handling is done by passing negative numbers.

    >>> def some_function(conn, *args):
    >>>     conn.send(0)
    >>>     a = 0
    >>>     try:
    >>>         for i in range(100):
    >>>                 a += 1
    >>>                 conn.send(i + 1)  # Send progress
    >>>     except Exception:
    >>>         conn.send(-1)  # Send error code
    >>>     else:
    >>>         conn.send(101)  # Send successful completion code

    Now we can use an instance of the ProgressDialog class within any 
    QtWidget to execute the operation in a separate process, show a progress 
    bar, and print the error code:

    >>> progress_dialog = ProgressDialog("Running...", some_function, self)
    >>> progress_dialog.finished.connect(lambda err_code: print(err_code))
    >>> progress_dialog.open()
    """

    def __init__(self, title, operation, args=(), parent=None):
        super().__init__(parent, Qt.WindowCloseButtonHint)
        self.setWindowTitle(title)
        self.progress_bar = QtWidgets.QProgressBar(self)
        self.progress_bar.setValue(0)
        layout = QtWidgets.QHBoxLayout()
        layout.addWidget(self.progress_bar)
        self.setLayout(layout)

        # Create connection pipeline
        self.parent_conn, self.child_conn = mp.Pipe()

        # Create process
        args = (self.child_conn, *args)
        self.process = mp.Process(target=operation, args=args)

        # Create status emitter
        self.progress_emitter = ProgressEmitter(self.parent_conn, self.process)
        self.progress_emitter.signals.progress.connect(self.slot_update_progress)
        self.thread_pool = QtCore.QThreadPool()

    def slot_update_progress(self, i):
        if i < 0:
            self.done(i)
        elif i == 101:
            self.done(0)
        else:
            self.progress_bar.setValue(i)

    def open(self):
        super().open()
        self.process.start()
        self.thread_pool.start(self.progress_emitter)

    def closeEvent(self, *args):
        self.progress_emitter.running = False
        self.process.terminate()
        super().closeEvent(*args)


class ProgressEmitter(QtCore.QRunnable):
    """Listens to status of process"""

    class ProgressSignals(QtCore.QObject):
        progress = QtCore.Signal(int)

    def __init__(self, conn, process):
        super().__init__()
        self.conn = conn
        self.process = process
        self.signals = ProgressEmitter.ProgressSignals()
        self.running = True

    def run(self):
        while self.running:
            if self.conn.poll():
                progress = self.conn.recv()
                self.signals.progress.emit(progress)
                if progress < 0 or progress == 101:
                    self.running = False
            elif not self.process.is_alive():
                self.signals.progress.emit(-999)
                self.running = False

[JonB]

@janhein_dejong
First, your real-life case may not be as bad as your tight-loop example.

Second, my machines are often UI-laggy if you do have a really heavy computation going on, doesn't matter whether it's separate processes or threads, c'est la vie!

Third, it gets complicated, but because you're using Python apparently multi-threading works such that only one thread at a time can use the Python interpreter. I don't know how this plays with a PyQt program, but it could mean your threads are effectively serialized.

And fourth & finally, again I don't know how that plays with Qt/PyQt, but I have read that from Python you are really supposed to use Python threading, not Qt threading.

See if you get a more sympathetic answer! You could Google a bit for Python/PyQt threading. If you don't get a better answer here, you could join the PyQt mailing list and ask there for any PyQt-specifics.

[JonB]

@Denni-0 said in Show QProgressbar with computationally heavy background process:

Next @JonB is incorrect in his statement about which Threading module you should use. If you are using Python-Qt you should use QThread as it works with the QApplication Event Handler better

Fair enough, you have more experience than I in this area.

@janhein_dejong
https://www.learnpyqt.com/courses/concurrent-execution/multithreading-pyqt-applications-qthreadpool/ looks up-to-date and is an interesting read. I think that is recommending QRunnable and the QThreadPool. There is also a discussion in https://stackoverflow.com/questions/6783194/background-thread-with-qthread-in-pyqt.

[janhein_dejong]

Thanks so much, I'll go ahead and implement these suggestions. @JonB what do you mean by LRP breaks?

[JonB]

@janhein_dejong said in Show QProgressbar with computationally heavy background process:

@JonB what do you mean by LRP breaks?

It was @Denni-0 who wrote "LRP breaks", not I. It will stand for "Long Running Processes". I think he means you will want to call https://doc.qt.io/qt-5/qcoreapplication.html#processEvents (which has its issues) periodically during a "LRP" in order to give the UI time to do stuff in order to keep it responsive.

[janhein_dejong]

Alright... I turned it into the example below. Already works much better... the tight loop is still somewhat laggy, but at least it reaches 100% when you expect it to. I'll try it on my real world example, to see how that behaves.

import sys
import time

from PySide2 import QtCore
from PySide2.QtCore import Qt
import PySide2.QtWidgets as QtWidgets


class MainWindow(QtWidgets.QMainWindow):
    """Main window, with one button for exporting stuff"""

    def __init__(self, parent=None):
        super().__init__(parent)
        central_widget = QtWidgets.QWidget(self)
        layout = QtWidgets.QHBoxLayout(self)
        button1 = QtWidgets.QPushButton("Tight operation")
        button2 = QtWidgets.QPushButton("Loose operation")
        button1.clicked.connect(self.export_tight_stuff)
        button2.clicked.connect(self.export_loose_stuff)
        layout.addWidget(button1)
        layout.addWidget(button2)
        central_widget.setLayout(layout)
        self.setCentralWidget(central_widget)

    def export_tight_stuff(self):
        """Opens dialog and starts exporting"""
        worker = Worker(self.tight_operation)
        some_window = MyExportDialog(worker, self)
        some_window.exec_()

    def export_loose_stuff(self):
        """Opens dialog and starts exporting"""
        worker = Worker(self.loose_operation)
        some_window = MyExportDialog(worker, self)
        some_window.exec_()

    @staticmethod
    def loose_operation():
        """Something that doesn't take a lot of CPU power"""
        time.sleep(.1)

    @staticmethod
    def tight_operation():
        """Something that takes a lot of CPU power"""
        some_val = 0
        for i in range(1_000_000):
            some_val += 1


class WorkerSignals(QtCore.QObject):
    progress = QtCore.Signal(int)
    finished = QtCore.Signal()


class Worker(QtCore.QRunnable):

    def __init__(self, fn):
        super().__init__()
        self.operation = fn
        self.signals = WorkerSignals()

    def run(self):
        cnt = 0
        while cnt < 100:
            cnt += 1
            self.operation()
            self.signals.progress.emit(cnt)
        self.signals.finished.emit()


class MyExportDialog(QtWidgets.QDialog):
    """Dialog which does some stuff, and shows it's progress"""

    def __init__(self, worker, parent=None):
        super().__init__(parent, Qt.WindowCloseButtonHint)
        self.setWindowTitle("Exporting...")
        layout = QtWidgets.QHBoxLayout()
        self.progress_bar = QtWidgets.QProgressBar(self)
        layout.addWidget(self.progress_bar)
        self.setLayout(layout)
        worker.signals.progress.connect(self.progress_bar.setValue)
        worker.signals.finished.connect(self.close)
        self.thread_pool = QtCore.QThreadPool()
        self.thread_pool.start(worker)


if __name__ == "__main__":
    app = QtWidgets.QApplication()
    window = MainWindow()
    window.show()
    sys.exit(app.exec_())

[janhein_dejong]

Hmm... good questions @Denni-0

Regarding super, do you mean it's better to use the syntax below? What are these dangers you speak of?

class SubClass: 

    def __init__(self, *args):
        SuperClass.__init__(self, *args)

Regarding sys.exit(app.exec_()) - the official Qt for Python page shows an example using that method. I'm using PySide2 (Qt for Python), so I guess for that binding that's still the way to go.

Regarding subclassing QThreadPool - I'm not sure I get your point completely. Should I subclass it? What's the advantage?

Regarding threading in Python - you're right I'm not entirely sure how this works internally. My understanding was that if you have two threads (A and B), the GIL will try to break these threads into smaller operations, and execute these chunks from both threads on the same core in your CPU sequentially. So in my example, the GIL breaks tight_operation into smaller pieces, and switches between the GUI thread, and the QThreadPool initiated thread between these smaller pieces; not solely upon completion of tight_operation. The GUI thread should therefore not be blocked by tight_operation. Am I right?

[janhein_dejong]

Ok... it took a bit of time, but I finally figured this one out.

The difficulty with showing a responsive user-interface while running a computationally heavy process using threading, stems from the fact in this case one combines a so-called IO-bound thread (i.e. the GUI), with a CPU-bound thread (i.e. the computation). For a IO-bound process, the time it takes to complete is defined by the fact that the thread has to wait on input or output (e.g. a user clicking on things, or a timer). By contrast, the time required to finish a CPU-bound process is limited by the power of the processing unit performing the process.

In principle, mixing these types of threads in Python should not be a problem. Although the GIL enforces that only one thread is running at a single instance, the operating system in fact splits the processes up into smaller instructions, and switches between them. If a thread is running, it has the GIL and executes some of its instructions. After a fixed amount of time, it needs to release the GIL. Once released, the GIL can schedule activate any other 'runnable' thread - including the one that was just released.

The problem however, is with the scheduling of these threads. Here things become a bit fuzzy for me, but basically what happens is that the CPU-bound thread seems to dominate this selection, from what I could gather due to a process called the "convey effect". Hence, the erratic and unpredictable behavior of a Qt GUI when running a CPU-bound thread in the background.

I found some interesting reading material on this:

• Understanding the GIL
• More in depth analysis of the GIL
• Nice visual representation of thread scheduling

So... this is very nice and all, how do we fix this?

In the end, I managed to get what I want using multiprocessing. This allows you to actually run a process parallel to the GUI, instead in sequential fashion. This ensures the GUI stays as responsive as it would be without the CPU-bound process in the background.

Multiprocessing has a lot of difficulties of its own, for example the fact that sending information back and forth between processes is done by sending pickled objects across a pipeline. However, the end-result is really superior in my case.

Below I put a code snippet, showing my solution. It contains a class called ProgressDialog, which provides an easy API for setting this up with your own CPU-bound process.

"""Contains class for executing a long running process (LRP) in a separate
process, while showing a progress bar"""

import multiprocessing as mp

from PySide2 import QtCore
from PySide2.QtCore import Qt
import PySide2.QtWidgets as QtWidgets


class ProgressDialog(QtWidgets.QDialog):
    """Dialog which performs a operation in a separate process, shows a
    progress bar, and returns the result of the operation

    Parameters
    ----
    title: str
        Title of the dialog
    operation: callable
        Function of the form f(conn, *args) that will be run
    args: tuple
        Additional arguments for operation
    parent: QWidget
        Parent widget

    Returns
    ----
    result: int
        The result is an integer. A 0 represents successful completion, or
        cancellation by the user. Negative numbers represent errors. -999
        is reserved for any unforeseen uncaught error in the operation.

    Examples
    ----
    The function passed as the operation parameter should be of the form
    ``f(conn, *args)``. The conn argument is a Connection object, used to
    communicate the progress of the operation to the GUI process. The
    operation can pass its progress with a number between 0 and 100, using
    ``conn.send(i)``. Once the process is finished, it should send 101.
    Error handling is done by passing negative numbers.

    >>> def some_function(conn, *args):
    >>>     conn.send(0)
    >>>     a = 0
    >>>     try:
    >>>         for i in range(100):
    >>>                 a += 1
    >>>                 conn.send(i + 1)  # Send progress
    >>>     except Exception:
    >>>         conn.send(-1)  # Send error code
    >>>     else:
    >>>         conn.send(101)  # Send successful completion code

    Now we can use an instance of the ProgressDialog class within any 
    QtWidget to execute the operation in a separate process, show a progress 
    bar, and print the error code:

    >>> progress_dialog = ProgressDialog("Running...", some_function, self)
    >>> progress_dialog.finished.connect(lambda err_code: print(err_code))
    >>> progress_dialog.open()
    """

    def __init__(self, title, operation, args=(), parent=None):
        super().__init__(parent, Qt.WindowCloseButtonHint)
        self.setWindowTitle(title)
        self.progress_bar = QtWidgets.QProgressBar(self)
        self.progress_bar.setValue(0)
        layout = QtWidgets.QHBoxLayout()
        layout.addWidget(self.progress_bar)
        self.setLayout(layout)

        # Create connection pipeline
        self.parent_conn, self.child_conn = mp.Pipe()

        # Create process
        args = (self.child_conn, *args)
        self.process = mp.Process(target=operation, args=args)

        # Create status emitter
        self.progress_emitter = ProgressEmitter(self.parent_conn, self.process)
        self.progress_emitter.signals.progress.connect(self.slot_update_progress)
        self.thread_pool = QtCore.QThreadPool()

    def slot_update_progress(self, i):
        if i < 0:
            self.done(i)
        elif i == 101:
            self.done(0)
        else:
            self.progress_bar.setValue(i)

    def open(self):
        super().open()
        self.process.start()
        self.thread_pool.start(self.progress_emitter)

    def closeEvent(self, *args):
        self.progress_emitter.running = False
        self.process.terminate()
        super().closeEvent(*args)


class ProgressEmitter(QtCore.QRunnable):
    """Listens to status of process"""

    class ProgressSignals(QtCore.QObject):
        progress = QtCore.Signal(int)

    def __init__(self, conn, process):
        super().__init__()
        self.conn = conn
        self.process = process
        self.signals = ProgressEmitter.ProgressSignals()
        self.running = True

    def run(self):
        while self.running:
            if self.conn.poll():
                progress = self.conn.recv()
                self.signals.progress.emit(progress)
                if progress < 0 or progress == 101:
                    self.running = False
            elif not self.process.is_alive():
                self.signals.progress.emit(-999)
                self.running = False