Python/PyQt QDialog with connect lambda "leaks"

[JonB]

I have posted the following to the PyQt mailing list. I wait to see whether I get a reply. If there are any PyQt users here who can answer, or you're a PyQt user and you'd like to know, please read and/or comment.

PyQt 5.7.

I have a large body of existing UI code. I have spent two days commenting in & out bits of code to try to discover why some of its QDialogs "leak" after calling QDialog.exec().

My definition of "leak" here is: after executing from somewhere else

dlg = QDialog(self)
QDialog.exec()

the instance of the dialog stays in existence permanently (as long as the caller exists, which for me is till end of program). That means that every time that code gets executed, yet another new dialog is left around in memory, which adds up over time. All I do to test is go into the dialog and immediately close it.

I discover this by inspecting QtWidgets.QApplication.allWidgets() and reporting all QDialogs which are still in existence. I see an ever-increasing number of these dialogs, one per each time it's constructed and executed, when & only when the code in the dialog is as follows.

I have finally tracked down the problematic line in the dialog's __init__(). Some of them have:

from elsewhere import ensureValidDecimal
self.lineEdit = QLineEdit(self)
self.lineEdit.editingFinished.connect(lambda: ensureValidDecimal(self))

The vital bit is: they connect() to a lambda which references self.

If the lambda does not need to pass self out as an argument, there will be no leak.

If I go define (in this case) in the dialog (I actually sub-class from all my QDialogs so I can add stuff) a dedicated function to avoid the lambda:

    def selfEnsureValidDecimal(self)
        ensureValidDecimal(self)

    self.lineEdit.editingFinished.connect(self.selfEnsureValidDecimal)

then there will also be no leak.

I can see that at some deep level there must be a reference counting issue here. In some shape or form, the fact that we have a lambda which passes self to the outside world must mean Python/PyQt wants to keep a reference to the dialog and this must be preventing its destruction.

But I don't know what to do about it. There is a lot of code with a lot of dialogs with all sorts of code attached. So I need some kind of explanation of what exactly can or cannot be done here, what to look for in code, etc. Note that I do not wish to use QDialog.setAttribute(QtCore.Qt.WA_DeleteOnClose, True) on all my dialogs (I believe that would solve the leak, but it's not the point). What must I not do if I do not expect such a self-reference to be left around preventing Python/PyQt from actually freeing up the dialog?

EDIT

Oh dear! Googling hard, I have just come across https://stackoverflow.com/a/48501804/489865, stating:

​Beware! As soon as you connect your signal to a lambda slot with a reference to self, your widget will not be garbage-collected! That's because lambda creates a closure with yet another uncollectable reference to the widget.
Thus, self.someUIwidget.someSignal.connect(lambda p: self.someMethod(p)) is very evil :)

which confirms exactly what I have spent days discovering and typing in :(

So, that is indeed the problem? And so I am required to go through all code looking for exactly the pattern to change it? Does this only apply to QWidget.connect(), I only need to look at all connect() calls, or are there other similar cases?

[kshegunov]

Well, my friend, as you often like to point out how C++ is too complicated now is it time for me to gloat? :)
Joking aside, is it possible that calling QObject::deleteLater after you've returned from the dialog would solve this particular conundrum?

[JonB]

@kshegunov
I have never defended Python! I don't have a choice for my Qt work, if I did I'd prefer C++. I just like C# or plain C.

deleteLater() is a possibility, requires code changing. It's not "one dialog" it's "the code [I didn't write] has loads & loads of dialogs, I have absolutely no idea which ones might have "connect-lambda-self"s (or maybe other idioms?) somewhere in them". If I deleteLater() after all of them (lots of places), seems I might just as well QDialog.setAttribute(QtCore.Qt.WA_DeleteOnClose, True) on all of them in initializer. That one worries because presumably then code must not be accessing any members of dialog after it's been closed, and I think some code does want to do that, and that would go wrong, right?

Sigh.

[kshegunov]

@JonB said in Python/PyQt QDialog with connect lambda "leaks":

That one worries because presumably then code must not be accessing any members of dialog after it's been closed, and I think some code does want to do that, and that would go wrong, right?

Right. I would prefer the deleteLater to the attribute, if it works at all, as you have a clear idea where is the deletion happening - just after you return control to the event loop.

[JonB]

@kshegunov
Yes, I agree, but that requires me to go through code finding all cases where a dialog is shown (e.g. exec()) and appending that line. And I still don't really know when it's "safe", because I don't know whether there is a later code path which requires to access the dialog's information after closure.

Since I haven't got time to do that, and since it's easy to do because I sub-class all my QDialogs via my own class, for now at least I have put QDialog.setAttribute(QtCore.Qt.WA_DeleteOnClose, True) into my derived class's constructor (if you can really call it that in Python, it's the __init__() method). Callers could then undo that immediately after construction if they want (i.e. it's just the default behaviour), if & when I ever get time to look through. I'll have to run with it and see....

None of this actually addresses the fundamental "closure-leakage" situation I describe, and I don't know if that principle could be applying in other cases in code. I knew there was a good reason why I never want to use lambdas in languages (at least Python)! Does C have them? No, never needed them. Does C#? Yes, in later versions, but I already have avoided them deliberately. Does C++ have them? Yes in newer versions, but then in practice C++ programs leak like a sieve anyway, so I don't expect much ;) "KISS".

Thanks for your input.

[kshegunov]

@JonB said in Python/PyQt QDialog with connect lambda "leaks":

Yes, I agree, but that requires me to go through code finding all cases where a dialog is shown (e.g. exec()) and appending that line. And I still don't really know when it's "safe", because I don't know whether there is a later code path which requires to access the dialog's information after closure.

If you ask me, find a better code 'cause this one seems very suspicious ... ;)

practice C++ programs leak like a sieve

Hey! My programs don't leak, and if they ever do it's a little drip, no gushing.

[JonB]

@kshegunov

If you ask me, find a better code 'cause this one seems very suspicious ... ;)

32K lines of existing code I inherited. Whadda you want me to do about it? ;)

Hey! My programs don't leak, and if they ever do it's a little drip, no gushing.

I do not doubt that you are (pretty) perfect :) However, have a look through the code posts on this forum of users' real-world Qt/C++ code and send me £1 for each time you spot where they have left a leak....!

[J.Hilk]

@JonB said in Python/PyQt QDialog with connect lambda "leaks":

32K lines of existing code I inherited. Whadda you want me to do about it? ;)

hopfuly it's at least decently structured and named/commented !?
Because I remember legacy code that was handed to me with things like 3000 lines inside the main() function and stuff.

Hey! My programs don't leak, and if they ever do it's a little drip, no gushing.

I do not doubt that you are (pretty) perfect :) However, have a look through the code posts on this forum of users' real-world Qt/C++ code and send me £1 for each time you spot where they have left a leak....!

I have to admit, early on in my days I had a slack stance on that as well. Usually with the mindset, "That class's instance will exist as long as the program is running anyway ".
Until it didn't. Lessen learend, eventually.

[JonB]

@J.Hilk said in Python/PyQt QDialog with connect lambda "leaks":

hopfuly it's at least decently structured and named/commented !?

Don't be silly! :)

[kshegunov]

@J.Hilk said in Python/PyQt QDialog with connect lambda "leaks":

Because I remember legacy code that was handed to me with things like 3000 lines inside the main() function and stuff.

The pinnacle of programming for me when I was handed over 60k+ fortran "scientific code" with not one function in it - everything in the main. The globals (i.e. common blocks) were several pages long and everything was "controlled" through infinitely long ifs. Sufficed to say, no comments were thought to be needed ... So it's never the language, it's the programmer (or lack thereof) that molds the code into life.

I have to admit, early on in my days I had a slack stance on that as well. Usually with the mindset, "That class's instance will exist as long as the program is running anyway ".

Leave water dropping on a piece of stone and it'd eat at it eventually ... it wouldn't matter if it's granite, basalt or sandstone, it would eventually succumb to the simple droplet. The moral is: If you're smart and have a self-preservation instinct, don't leak stuff! :)

[JonB]

@kshegunov , @J-Hilk
Don't tell anybody I said so, but this perfectly legitimate wikipedia link is actually my favorite language.

It doesn't mess about allowing any such thing as comments in the language in any shape or form, and the standard Hello World code is:

++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.

Programmers are too soft these days....

[J.Hilk]

@JonB said in Python/PyQt QDialog with connect lambda "leaks":

Programmers are too soft these days....

I'm just glad, that the EDSAC days are over :-)

[JonB]

For the original problem, and https://stackoverflow.com/a/48501804/489865, stating "a connected lambda which references self will not be garbage collected", I received the following code function from the PyQt mailing list. I am not going to use it myself unless I have to (I prefer to rewrite code so as not to use self-referencing lambdas), but I paste it here (untested; and the author states it's for Python 2.7 but should work for Python 3) in case other readers in future might be interested.

def connect_lambda(bound_signal, self, func, **kw):
    # It can be used like this:
    # Instead of
    #   self.editingFinished.connect(lambda: self.whatever())
    # do
    #   connect_lambda(self.editingFinished, self, lambda self: self.whatever())

    import weakref
    r = weakref.ref(self)
    del self
    num_args = func.__code__.co_argcount - 1
    if num_args < 0:
        raise TypeError('lambda must take at least one argument')

    def slot(*args):
        ctx = r()
        if ctx is not None:
            if len(args) != num_args:
                args = args[:num_args]
            func(ctx, *args)

    bound_signal.connect(slot, **kw)