PySide6 memory model and QObject lifetime management

[mliberty]

I have authored and maintained a number of Python/Qt applications over a number of years. I thought that I was familiar with the memory model challenges between Qt C++ and PySide6 Python. However, I am seeing very infrequent crashes in the Joulescope UI that are not consistent across platforms, which may indicate improper memory management. I am now questioning everything that I thought I knew about Qt6 / PySide6 / Qt for Python memory management.

I searched through the Qt for Python document, but I did not see any definitive section about proper memory management. Did I miss something? Any authoritative references are appreciated. There are lots of blogs and posts, but I am specifically looking for PySide6 (not PyQt5, PyQt6, PySide2). This recent forum post is very related.

The official examples definitely show holding Python object references as member variables of the parent.

The C++ objects use explicit memory management with new/delete. Qt helps by managing the parent/child relationships and automatically deleting all children when a parent is deleted. In Python, memory management is automatic by variable scope using reference counting.

My mental model is that you have to maintain a reference to the Python-side object for the lifecycle of the underlying C++ object. The general way to do this is to store each child widget as a Python instance member.

My mental model is also that signal connections maintain a reference to the connected "slot" callable.

However, I put together a very simple example that disproves my mental model:

from PySide6 import QtCore, QtGui, QtWidgets
import sys
import gc


class MainWindow(QtWidgets.QMainWindow):

    signal1 = QtCore.Signal(str)
    signal2 = QtCore.Signal(str)

    def __init__(self, parent=None):
        super().__init__(parent)
        fn = lambda title: self.slot1(title)
        fn_before = sys.getrefcount(fn)
        self.signal1.connect(fn)
        fn_after = sys.getrefcount(fn)
        print(f'Lambda: refcount_incr={fn_after - fn_before}, type={type(fn)}')
        
        self_before = sys.getrefcount(self)
        slot2_before = sys.getrefcount(MainWindow.slot2)
        bound_method = self.slot2
        bound_method_before = sys.getrefcount(fn)
        self.signal2.connect(bound_method)
        bound_method_after = sys.getrefcount(fn)
        print(f'Bound method: refcount_incr={bound_method_after - bound_method_before}, type={type(bound_method)}')
        del bound_method
        slot2_after = sys.getrefcount(MainWindow.slot2)
        self_after = sys.getrefcount(self)
        print(f'slot2: refcount_incr={slot2_after - slot2_before}')
        print(f'self: refcount_incr={self_after - self_before}')
        
        self._widget = QtWidgets.QWidget(self)
        self.setCentralWidget(self._widget)
        self._layout = QtWidgets.QVBoxLayout(self._widget)
        label = QtWidgets.QLabel('hello')
        label_before = sys.getrefcount(label)
        self._layout.addWidget(label)
        label_after = sys.getrefcount(label)
        print(f'Label: refcount_incr={label_after - label_before}, type={type(label)}')
        gc.collect()
        gc.collect()
        print(f'object count = {len(gc.get_objects())}')
    
    @QtCore.Slot(str)
    def slot1(self, txt):
        print(f'slot1 received: {txt}')

    @QtCore.Slot(str)
    def slot2(self, txt):
        print(f'slot2 received: {txt}')


if __name__ == '__main__':
    app = QtWidgets.QApplication([])
    w = MainWindow()
    gc.collect()
    gc.collect()
    print(f'object count = {len(gc.get_objects())}')
    w.signal1.emit('hello')
    w.signal2.emit('world')

When I run on Windows 11 with Python 3.11.8 64-bit and PySide 6.6.1, I see:

Lambda: refcount_incr=1, type=<class 'function'>
Bound method: refcount_incr=0, type=<class 'method'>
slot2: refcount_incr=0
self: refcount_incr=0
Label: refcount_incr=1, type=<class 'PySide6.QtWidgets.QLabel'>
object count = 56872
object count = 56872
slot1 received: hello
slot2 received: world

Two things are different than expected. I appreciate your help and feedback to better understand what is happening!

1. Adding the label to the layout increments its reference count. This means that QWidgets do not need to be also referenced using Python object members. Have I been unnecessarily worried about ensuring all QWidgets maintain Python references? Thinking back over many years, I have seen problems with both C++ objects being deleted and Python objects being deleted. Is managing child widget lifetimes now automatic in PySide6?

2. Connecting a lambda to a signal increments the lambda's refcount, so the lambda should remain valid. However, connecting a bound method to a signal does not increment the bound method's refcount. The bound method will go out of scope and should be garbage collected. The unbound class method refcount and self refcount does not change either, so I don't think the PySide6 layer is storing any reference. Therefore, I expect that the signal now has an callable with refcount=0 that may trigger a segfault. However, this is not happening in my trivial example despite running the Python garbage collector. Is connecting a bound method to a signal allowed? Do we also need to maintain a separate reference to the bound method (such as by using a lambda)?

I started diving into the source code for qobjectConnect, SignalManager::registerMetaMethodGetIndex and PyObject_PTR_CppToPython_PyObject which performs the Py_INCREF, but it will take me quite some time to understand the shiboken & wrapper details. Thank you in advance for your help!

[mliberty]

The Shiboken documentation includes a page on Object ownership.

The Parent-Child Relationship section describes how Shiboken automatically infers parent-child relationships. This documentation addresses (1). Yes, child objects are managed automatically by Shiboken. The label reference count increment is expected and we can rely on it. The parent QWidget (no longer?) needs to maintain a reference to all child QWidgets in Python, only the ones it wants to later manipulate.

I performed some more testing on signal connections with every callable type I could think of:

from PySide6 import QtCore, QtGui, QtWidgets
import sys
import gc


def connect(name, signal, slot_callable):
    refcount_before = sys.getrefcount(slot_callable)
    signal.connect(slot_callable)
    refcount_after = sys.getrefcount(slot_callable)
    print(f'{name}: refcount_incr={refcount_after - refcount_before}, type={type(slot_callable)}')


def function(txt):
    print(f'function: {txt}')


class MainWindow(QtWidgets.QMainWindow):

    signal1 = QtCore.Signal(str)

    def __init__(self, parent=None):
        super().__init__(parent)
        connect('function', self.signal1, function)
        lambda_fn = lambda title: self.slot1(title)
        connect('lambda', self.signal1, lambda_fn)
        connect('static', self.signal1, MainWindow.static_fn)
        connect('class', self.signal1, MainWindow.class_fn)
        connect('bound1', self.signal1, self.slot1)
        connect('bound2', self.signal1, self.slot2)
        connect('instance', self.signal1, self)

    @QtCore.Slot(str)
    @staticmethod
    def static_fn(txt):
        print(f'static: {txt}')

    @QtCore.Slot(str)
    @classmethod
    def class_fn(clz, txt):
        print(f'class: {txt}')
    
    @QtCore.Slot(str)  # with slot decorator
    def slot1(self, txt):
        print(f'slot1 received: {txt}')

    def slot2(self, txt):  # no slot decorator
        print(f'slot2 received: {txt}')
        
    def __call__(self, txt):
        print(f'instance received: {txt}')


if __name__ == '__main__':
    app = QtWidgets.QApplication([])
    w = MainWindow()
    gc.collect()
    gc.collect()
    w.signal1.emit('hello')

When I run on Windows 11 with Python 3.11.8 64-bit and PySide 6.6.1, I see:

function: refcount_incr=1, type=<class 'function'>
lambda: refcount_incr=1, type=<class 'function'>
static: refcount_incr=1, type=<class 'function'>
class: refcount_incr=0, type=<class 'method'>
bound1: refcount_incr=0, type=<class 'method'>
bound2: refcount_incr=0, type=<class 'method'>
instance: refcount_incr=1, type=<class '__main__.MainWindow'>
function: hello
slot1 received: hello
static: hello
class: hello
slot1 received: hello
slot2 received: hello
instance received: hello

This is very interesting. The refcount increments for function and instance types, but not for method types. Is this intentional?

[mliberty]

I created a simple example that demonstrates the different behavior between connecting a signal to a lambda and connecting a signal to a bound method:

from PySide6 import QtCore, QtGui, QtWidgets
import sys


class Receiver(QtCore.QObject):

    def __del__(self):
        print('Receiver.__del__')

    @QtCore.Slot(str)
    def slot1(self, txt):
        print(f'slot1 received: {txt}')


class MainWindow(QtWidgets.QMainWindow):

    signal1 = QtCore.Signal(str)

    def __init__(self, parent=None):
        super().__init__(parent)
        rx = Receiver(self)  # passing self keeps rx valid through parent-child relationship
        self.signal1.connect(rx.slot1)  # does not work, rx refcount goes to 0 when exit __init__
        # self.signal1.connect(lambda txt: rx.slot1(txt))  # works, lambda keeps rx ref
        self.signal1.emit('rx in __init__ scope')


if __name__ == '__main__':
    app = QtWidgets.QApplication([])
    w = MainWindow()
    w.signal1.emit('SUCCESS')
    print('done')

When connecting to a bound method, the signal.connect() call does not increment the reference count on the bound method. It goes out of scope and future calls fail. On Windows for this example, the failure is silent with no crash.

To test the lambda, comment out the first self.signal1.connect line and uncomment the next.

When connecting to the lambda, the lambda's closure maintains a reference to rx and the signal.connect() increments the lambda's refcount, so both the lambda and the rx instance remain valid.

Any thoughts as to whether signal connect not incrementing the refcount of bound methods is a designed feature or a bug?

[JonB]

@mliberty
I'm jumping in here with a couple of possibilities.

When connecting to the lambda, the lambda's closure maintains a reference to rx and the signal.connect() increments the lambda's refcount, so both the lambda and the rx instance remain valid.

And, speaking from PyQt5 at least which probably applies to PySide too, don't you find that once connected to a lambda that will mean the whole instance will never get released by Python? Because it always has a refcount from the anonymous lambda. I recall a stackoverflow about this many years ago. Yuck for memory management.

Any thoughts as to whether signal connect not incrementing the refcount of bound methods is a designed feature or a bug?

Just maybe this allows the instance to be properly freed, by not incrementing the count per the previous point? Because I believe connecting to function slots does get freed in usual circumstances.

Wild hunches :)

[mliberty]

@JonB Thanks! I tend to think of slots as just any callable for PySide6 since any callable "just works".
For small applications, you can get away with this mental model. For larger applications, the memory management implications appear to need much closer attention.

Connecting Signals to Slots is a non-ownership relationship in C++. ~QObject automatically disconnects all signals and slots for the destroyed instance. You can manually use disconnect for more dynamic control. The notion of Signal/Slot connections having Python ownership of the Slot callable makes things ugly quickly from a memory management perspective. Refcount alone will not free the circular references, and you will eventually need the mark & sweep garbage collector to free them, if that ends up being possible at all as you pointed out.

For pure functions and staticmethods, ownership normally doesn't matter since their lifetimes are often the life of the application. If you want to unload/reload their module, then it's a problem. However, as you point out, lambdas really need to be owned by Signal.connect(), but then you can never disconnect unless you separately and manually maintain a reference.

To make things more confusing, for the same instance and method, the bound method instance will change. Python performs method binding dynamically at runtime, so calling print(f'{id(obj.foo)}') may return different values at different times. This must be why I do not see the bound method refcount increment. PySide6 must be binding at the C++ object level. It also must make internal C++ slots to wrap generic Python callables. This makes sense but makes memory management confusing unless you are using real Slots.

With this perspective, bound method refcounts should not be incremented by connect. I now see that attempting to use Python lambdas as a slot for anything that does not exist for the entire application lifetime is a recipe for pain.

Back to my original problem with crashes: it is not from using real Slots since Slots will be properly and automatically disconnected along with the object delete using deleteLater. With proper use, Qt is designed to ensure no Slot invocations for destroyed objects. However, lambdas can definitely be a problem since the closure will keep the Python instance alive after the C++ object is deleted. Time to justify or remove the 67 instances of .connect(lambda from the code!

[SGaist]

Hi,

This reminds me of the C++ 4 arguments connect variant that shall be used when using lambda. The third argument is a context object which when destroyed will trigger the deletion of the lambda object associated.

[mliberty]

Thanks @SGaist! I just looked, and Qt can use C++ lambdas / functors in this way. Here is the C++ example from the QtCore Signals & Slots page:

connect(sender, &QObject::destroyed, this, [=](){ this->m_objects.remove(sender); });

In both these cases, we provide this as context in the call to connect(). The context object provides information about in which thread the receiver should be executed. This is important, as providing the context ensures that the receiver is executed in the context thread.
The lambda will be disconnected when the sender or context is destroyed. You should take care that any objects used inside the functor are still alive when the signal is emitted.*

Unfortunately, it looks like PySide 6.6.1 does not include this signature:

TypeError: 'PySide6.QtCore.QObject.connect' called with wrong argument types:
  PySide6.QtCore.QObject.connect(MainWindow, bytes, Receiver, function)
Supported signatures:
  PySide6.QtCore.QObject.connect(PySide6.QtCore.QObject, bytes, Callable, PySide6.QtCore.Qt.ConnectionType = Instance(Qt.AutoConnection))
  PySide6.QtCore.QObject.connect(bytes, Callable, PySide6.QtCore.Qt.ConnectionType = Instance(Qt.AutoConnection))
  PySide6.QtCore.QObject.connect(bytes, PySide6.QtCore.QObject, bytes, PySide6.QtCore.Qt.ConnectionType = Instance(Qt.AutoConnection))
  PySide6.QtCore.QObject.connect(PySide6.QtCore.QObject, bytes, bytes, PySide6.QtCore.Qt.ConnectionType = Instance(Qt.AutoConnection))
  PySide6.QtCore.QObject.connect(PySide6.QtCore.QObject, PySide6.QtCore.QMetaMethod, PySide6.QtCore.QObject, PySide6.QtCore.QMetaMethod, PySide6.QtCore.Qt.ConnectionType = Instance(Qt.AutoConnection))
  PySide6.QtCore.QObject.connect(PySide6.QtCore.QObject, bytes, PySide6.QtCore.QObject, bytes, PySide6.QtCore.Qt.ConnectionType = Instance(Qt.AutoConnection))

I think we would need:

PySide6.QtCore.QObject.connect(PySide6.QtCore.QObject, bytes, PySide6.QtCore.QObject, Callable, PySide6.QtCore.Qt.ConnectionType = Instance(Qt.AutoConnection))

This would make Python lambdas possible, but you still would have to be very careful about the closure references. Simplified, this is
connect(QObject, bytes, QObject, Callable, ConnectionType=Auto)
The three-argument version is available, but not with the extra Slot QObject. I wonder why?

[JonB]

@mliberty
I don't understand. The second QObject is the slot object.

https://doc.qt.io/qtforpython-6/PySide6/QtCore/QObject.html#PySide6.QtCore.PySide6.QtCore.QObject.connect

static PySide6.QtCore.QObject.connect(sender, signal, receiver, method[, type=Qt.AutoConnection])

is the one accepting a lambda, with a slot object specified so lambda can be released. No?

[mliberty]

@JonB I totally agree that the third argument is a QObject that is the Slot context. What I don't know is how to get from a Python lambda (python function type) to a QMetaMethod.

I can get from the signal to the QMetaMethod, like this:

signal_idx = self.metaObject().indexOfProperty("signal1")
signal = self.metaObject().property(signal_idx).notifySignal()

My searching has not yielded any good results for the lambda, though.

The other 4 argument method takes (QObject, bytes, QObject, bytes) which won't work either.

[mliberty]

@JonB Interestingly, I see this signature in the documentation you referenced:

static PySide6.QtCore.QObject.connect(sender, signal, context, functor[, type=Qt.AutoConnection])
PARAMETERS:
sender – PySide6.QtCore.QObject
signal – str
context – PySide6.QtCore.QObject
functor – PyCallable
type – ConnectionType

but it's not appearing in the list I see in error report.

And here is why:

Added last week!

[JonB]

@mliberty
Yes, that one looks just right. Can you call PySide6.QtCore.QObject.connect() just like that, with a lambda for functor, or do you have to go through the Python/PySide object.signal.connect(lambda: ... ) syntax, with nowhere for the slot object? Which I think is what you are wanting.

The rest of it beyond me now.

[mliberty]

I did some additional testing, the the refcount for a Python lambda used as a slot is not automatically decremented when a QObject is deleted. I tried PySide 6.6.2 which includes the correct 4 argument connect function. I could not find a way of passing a QObject that would result in the lambda refcount being decremented to 0 so that it would be deleted. This could potentially be considered a PySide6 bug, but I am not sure passing a lambda as a slot is truly valid, either.

I have found a different approach that both isolates the lambda closure and allows for automatic delete. Instead of directly connecting the lambda to the Signal, the approach uses a separate QObject wrapper:

class CallableSlotAdapter(QtCore.QObject):
    """A QObject that calls a python callable whenever its slot is called.

    :param parent: The required parent QObject that manages this
        instance lifecycle through the Qt parent-child relationship.
    :param fn: The Python callable to call.

    Connect the desired signal to self.slot.
    """

    def __init__(self, parent, fn):
        self._fn = fn
        super().__init__(parent)

    def slot(self, *args):
        code = self._fn.__code__
        co_argcount = code.co_argcount
        if args and isinstance(self._fn, types.MethodType):
            args -= 1
        self._fn(*args[:co_argcount])

Here is an example for how to use it:

button = QtWidgets.QPushButton('Push me', parent)
adapter = CallableSlotAdapter(button, lambda checked=False: print(f'pressed {checked}')
button.pressed.connect(adapter.slot)

Since a large percentage of my lambda usage was for QAction in QMenu, I also made this:

class CallableAction(QtGui.QAction):
    """An action that calls a python callable.

    :param parent: The parent for this action.  If parent is a QMenu instance,
        or QActionGroup who has a QMenu instance parent,
        then automatically call parent.addAction(self).
    :param text: The text to display for the action.
    :param fn: The callable that is called with each trigger.
    :param checkable: True to allow checkable.
    :param checked: Set the checked state, if checkable.

    As of Feb 2024, connecting a callable to a signal increments
    the callable's reference count.  Unfortunately, this reference
    count is not decremented when the object is deleted.

    This class provides a Python-side wrapper for a QAction that
    connects to a Python callable.  You can safely provide a lambda
    which will be dereferenced and deleted along with the QAction.

    For a discussion on PySide6 memory management, see
    https://forum.qt.io/topic/154590/pyside6-memory-model-and-qobject-lifetime-management/11
    """

    def __init__(self, parent: QtCore.QObject, text: str, fn: callable, checkable=False, checked=False):
        self._fn = fn
        super().__init__(text, parent=parent)
        if bool(checkable):
            self.setCheckable(True)
            self.setChecked(bool(checked))
        self.triggered.connect(self._on_triggered)
        while isinstance(parent, QtGui.QActionGroup):
            parent = parent.parent()
        if isinstance(parent, QtWidgets.QMenu):
            parent.addAction(self)

    def _on_triggered(self, checked=False):
        code = self._fn.__code__
        args = code.co_argcount
        if args and isinstance(self._fn, types.MethodType):
            args -= 1
        if args == 0:
            self._fn()
        elif args == 1:
            self._fn(checked)

Since we can rely on QObject parent-child relationship to manage the child object lifetime, both CallableSlotAdapter and CallableAction can correctly decrement the fn refcount and delete the fn that is a lambda, assuming no one else maintains a reference to the lambda.

Here is my example code that shows that this works:

from PySide6 import QtCore, QtGui, QtWidgets
import sys
import gc
import html
import weakref


def _deref(name, obj):
    if obj is None:
        return f'{name}: no weakref yet'
    obj = obj()
    if obj is None:
        return f'{name}: weakref None'
    return f'{name}: refcount={sys.getrefcount(obj) - 2} | {obj}'


class CallableAction(QtGui.QAction):

    def __init__(self, parent: QtCore.QObject, text: str, fn: callable):
        self._fn = fn
        super().__init__(text, parent)
        self.triggered.connect(self._on_triggered)

    def _on_triggered(self):
        self._fn()


class MainWindow(QtWidgets.QMainWindow):

    tick_value = QtCore.Signal(str)

    def __init__(self, parent=None):
        self._menu = None
        self._action = None
        self._fn = None
        super().__init__(parent)
        self._widget = QtWidgets.QWidget(self)
        self.setCentralWidget(self._widget)
        self._layout = QtWidgets.QVBoxLayout(self._widget)
        
        label = QtWidgets.QLabel('Hello world')
        label.setToolTip('my tooltip')
        self.tick_value.connect(label.setText)
        self._layout.addWidget(label)
        self._label_wref = weakref.ref(label)
        
        custom_label = QtWidgets.QLabel('Another label')
        self._layout.addWidget(custom_label)

        button = QtWidgets.QPushButton('Push me')
        button.pressed.connect(self._on_button)
        self._layout.addWidget(button)
        
        self._status = QtWidgets.QLabel()
        self._layout.addWidget(self._status)
        
        self._tick = 0
        self._on_timer()
        self._timer = QtCore.QTimer(self)
        self._timer.timeout.connect(self._on_timer)
        self._timer.start(1000)
        
    @QtCore.Slot()
    def _on_timer(self):
        gc.collect()
        gc.collect()
        s = [str(self._tick)]
        self._tick += 1
        s.append(html.escape(_deref('menu', self._menu)))
        s.append(html.escape(_deref('action', self._action)))
        s.append(html.escape(_deref('lambda', self._fn)))
        s.append(html.escape(_deref('label', self._label_wref)))
        s = '<html><body>' + '<br/>'.join(s) + '</body></html>'
        self.tick_value.emit(s)
    
    @QtCore.Slot()
    def _on_button(self):
        self._status.setText('button pressed')
    
    @QtCore.Slot()
    def _on_section1_1(self):
        self._status.setText('section1_1')
        
    @QtCore.Slot()
    def _on_section1_2(self):
        self._status.setText('section1_2')
    
    def _on_section1_3(self):
        self._status.setText('section1_3')
    
    def mousePressEvent(self, event):
        event.accept()
        if event.button() == QtCore.Qt.RightButton:
            menu = QtWidgets.QMenu('My menu', self)  # must set parent!
            menu.triggered.connect(menu.deleteLater)
            s1 = menu.addMenu('Section 1')
            
            # Manually add an action, set action as parent
            a1_1 = QtGui.QAction('Item 1', s1)  # must set parent!
            a1_1.triggered.connect(self._on_section1_1)
            s1.addAction(a1_1)

            # Simple action add
            s1.addAction('Item 2', self._on_section1_2)
            
            # Add an action that uses a lambda function as a slot
            # The lambda closure contains self, which outlives the menu
            # On delete, slots should be disconnected
            fn = lambda: print(f'lambda dir={dir()}') or self._on_section1_3()
            self._fn = weakref.ref(fn)
            #refcount_before = sys.getrefcount(fn)
            #a1_3 = s1.addAction('Item 3', fn)
            #refcount_after = sys.getrefcount(fn)
            #print(f'fn refcount incr = {refcount_after - refcount_before}')
            a1_3 = CallableAction(s1, 'Item 3', fn)
            s1.addAction(a1_3)
            self._action = weakref.ref(a1_3)
            
            # Display context menu at mouse press location
            menu.popup(event.globalPosition().toPoint())    
            self._menu = weakref.ref(menu)
        
        
if __name__ == '__main__':
    app = QtWidgets.QApplication([])
    w = MainWindow()
    w.show()
    app.exec()

After launching, right-click to open the context menu, which uses weakrefs to snoop the object lifecycle.

I updated my codebase to use this approach. I am also scouring the code to make sure that it speciifies a parent either indirectly through QLayout.addWidget or directly in the constructor (e.g. QAction, QTimer) for all QObjects. I found a few places keeping Python QAction references without assigning a parent. Combined with the improved Slot lambda handling, it should eliminate memory leaks and method invocations on potentially deleted objects.

Here is my summary of how to correctly manage memory for PySide6 / Qt6:

1. Ensure all QObjects get a parent, except for the very few top-level QObjects. You only need to maintain a Python reference to the QObjects without a parent. You may opt to keep a reference to QObjects you need to manipulate through code and not just Signal / Slot connections.
2. When using Signals and Slots, any callable works as a slot in Python. Avoid the temptation and stick to QObjects with Slot methods. Qt is designed for this.
3. If you do want to use a lambda for a Slot, use a wrapper QObject with a Slot method that invokes the lambda.
4. Note that Qt automatically deletes all children when the parent is deleted.
5. Note that Qt automatically disconnects all Signals and Slots for a QObject when that QObject is deleted.
6. Use QObject.deleteLater() to delete QObjects.
7. Keep QObject Python references local. Use Signals and Slots to communicate, especially between parent-child Widgets. For larger applications, Signals & Slots communication has scaling challenges. Consider using PubSub.