Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. Qt for Python
  4. How to debug a hard to find memory leak in PyQt

How to debug a hard to find memory leak in PyQt

Scheduled Pinned Locked Moved Unsolved Qt for Python
16 Posts 3 Posters 4.6k Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J JonB
    29 Oct 2020, 08:13

    @argosopentech
    Since this seems to be a PyQt5 issue, have you tried the folks over at https://riverbankcomputing.com/mailman/listinfo/pyqt, by joining that mailing list and asking your question? The author of PyQt5 is there. I don't know whether they will take the time to examine your code/issue, but it's where I would try.

    A Offline
    A Offline
    argosopentech
    wrote on 29 Oct 2020, 13:17 last edited by
    #6

    @JonB Thanks for the suggestion I just emailed.

    J 1 Reply Last reply 29 Oct 2020, 13:49
    0
    • A argosopentech
      29 Oct 2020, 13:17

      @JonB Thanks for the suggestion I just emailed.

      J Offline
      J Offline
      JonB
      wrote on 29 Oct 2020, 13:49 last edited by JonB
      #7

      @argosopentech
      I have seen your post. I don't know how people there will react to you only referring to this post here --- they like the information in their own forum posts. If it were me I might post a follow-up (i.e. Reply to all) there yourself in which you include the first, second & third of your posts here above.

      A 1 Reply Last reply 29 Oct 2020, 22:23
      1
      • J JonB
        29 Oct 2020, 13:49

        @argosopentech
        I have seen your post. I don't know how people there will react to you only referring to this post here --- they like the information in their own forum posts. If it were me I might post a follow-up (i.e. Reply to all) there yourself in which you include the first, second & third of your posts here above.

        A Offline
        A Offline
        argosopentech
        wrote on 29 Oct 2020, 22:23 last edited by
        #8

        @JonB Thanks for the suggestion I just replied with more information.

        J 1 Reply Last reply 30 Oct 2020, 15:22
        0
        • A argosopentech
          29 Oct 2020, 22:23

          @JonB Thanks for the suggestion I just replied with more information.

          J Offline
          J Offline
          JonB
          wrote on 30 Oct 2020, 15:22 last edited by JonB
          #9

          @argosopentech
          I see that you have just received a reply from mailing list :) And that guy is the PyQt5 author, like I said, and probably knows exactly what he is talking about! :) He is not very chatty, just terse and to-the-point, you just have to try to act as best as you can on what he tells you.

          1 Reply Last reply
          1
          • A Offline
            A Offline
            argosopentech
            wrote on 30 Oct 2020, 22:08 last edited by
            #10

            @JonB thanks for the help, I'm looking at his suggestions now. Here's his response for anyone curious:

            I don't see how the above can be expected to work, no matter what the
            run() method is doing. Your WorkerThread objects are likely to be
            garbage collected before they are finished and the del won't protect
            the thread.

            Try making the GUIWindow the parent of the WorkerThreads and see if that
            makes a difference.

            Phil

            1 Reply Last reply
            0
            • A Offline
              A Offline
              argosopentech
              wrote on 31 Oct 2020, 00:18 last edited by
              #11

              My understanding of what he's saying is that because the QThread is being managed by Qt blocking in the Python __del__ function isn't going to protect your QThread from being deleted while you're still using it. This didn't seem to be a problem I was having, I've been getting memory leaks not crashes from the QThreads being prematurely deleted though maybe them getting cleaned up early is leading to other memory being leaked. I based my original code on a tutorial I found on PyQt QThreads that uses the __del__ function this way but it makes sense that it isn't a great way of doing things.

              I tried making the QMainWindow the parent of the QThread like he suggested but that didn't seem to fix the problem. I also connected the QThread's finished signal with its deleteLater slot in line with Qt's documentation's example of subclassing QThread.

              from PyQt5.QtWidgets import QMainWindow, QApplication
              from PyQt5.QtCore import QThread
              import ctranslate2
              
              class WorkerThread(QThread):
                  def run(self):
                      translator = ctranslate2.Translator('/path/to/ctranslate/model')
              
              class GUIWindow(QMainWindow):
                  def translate(self):
                      new_worker_thread = WorkerThread(self)
                      new_worker_thread.finished.connect(new_worker_thread.deleteLater)
                      new_worker_thread.start()
              
              app = QApplication([])
              main_window = GUIWindow()
              main_window.show()
              
              for i in range(120):
                  print(i)
                  main_window.translate()
              
              app.exec_()
              
              

              I'm going to look into his suggestion more but I'm not sure it solves the issue. The other example in the Qt documentation puts the work in a worker object and pass to a QThread using moveToThread so I'm going to try structuring the threading like that and see what happens.

              J 1 Reply Last reply 31 Oct 2020, 09:19
              0
              • A argosopentech
                31 Oct 2020, 00:18

                My understanding of what he's saying is that because the QThread is being managed by Qt blocking in the Python __del__ function isn't going to protect your QThread from being deleted while you're still using it. This didn't seem to be a problem I was having, I've been getting memory leaks not crashes from the QThreads being prematurely deleted though maybe them getting cleaned up early is leading to other memory being leaked. I based my original code on a tutorial I found on PyQt QThreads that uses the __del__ function this way but it makes sense that it isn't a great way of doing things.

                I tried making the QMainWindow the parent of the QThread like he suggested but that didn't seem to fix the problem. I also connected the QThread's finished signal with its deleteLater slot in line with Qt's documentation's example of subclassing QThread.

                from PyQt5.QtWidgets import QMainWindow, QApplication
                from PyQt5.QtCore import QThread
                import ctranslate2
                
                class WorkerThread(QThread):
                    def run(self):
                        translator = ctranslate2.Translator('/path/to/ctranslate/model')
                
                class GUIWindow(QMainWindow):
                    def translate(self):
                        new_worker_thread = WorkerThread(self)
                        new_worker_thread.finished.connect(new_worker_thread.deleteLater)
                        new_worker_thread.start()
                
                app = QApplication([])
                main_window = GUIWindow()
                main_window.show()
                
                for i in range(120):
                    print(i)
                    main_window.translate()
                
                app.exec_()
                
                

                I'm going to look into his suggestion more but I'm not sure it solves the issue. The other example in the Qt documentation puts the work in a worker object and pass to a QThread using moveToThread so I'm going to try structuring the threading like that and see what happens.

                J Offline
                J Offline
                JonB
                wrote on 31 Oct 2020, 09:19 last edited by JonB
                #12

                @argosopentech said in How to debug a hard to find memory leak in PyQt:

                like he suggested but that didn't seem to fix the problem

                I suggest you go back with this fix this then, and ask him very politely to have a look and see if he can suggest anything else. Throw yourself at his mercy, nicely :) (But do try anything else you can think of from his suggestion before doing so.)

                1 Reply Last reply
                0
                • S Offline
                  S Offline
                  SGaist
                  Lifetime Qt Champion
                  wrote on 31 Oct 2020, 09:28 last edited by
                  #13

                  Do you really need to create a new translator each time ?

                  I am wondering whether you should reconsider your architecture.

                  It looks like you could make use of QtConcurrent to manage translation tasks rather than doing your own thread management.

                  Interested in AI ? www.idiap.ch
                  Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                  1 Reply Last reply
                  0
                  • A Offline
                    A Offline
                    argosopentech
                    wrote on 31 Oct 2020, 14:10 last edited by
                    #14

                    @JonB thanks that's what I was thinking too. I just wanted to post here first to make sure I wasn't missing something obvious or misunderstanding him before I sent another email.

                    @SGaist This was something that was mentioned on the OpenNMT forum thread I started too. It seems like not making a new Translator every time would be good for performance too. I'm going to try this and I think it should at least drastically slow down my memory leak. However, since users can switch between translations some will still need to be garbage collected so I'd like to figure out what's causing this to leak.

                    1 Reply Last reply
                    0
                    • S Offline
                      S Offline
                      SGaist
                      Lifetime Qt Champion
                      wrote on 31 Oct 2020, 14:44 last edited by
                      #15

                      Switching to a different language should be part of your API rather than a constraint. That way you can reload or replace the translator when appropriate.

                      Interested in AI ? www.idiap.ch
                      Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                      A 1 Reply Last reply 31 Oct 2020, 17:23
                      1
                      • S SGaist
                        31 Oct 2020, 14:44

                        Switching to a different language should be part of your API rather than a constraint. That way you can reload or replace the translator when appropriate.

                        A Offline
                        A Offline
                        argosopentech
                        wrote on 31 Oct 2020, 17:23 last edited by
                        #16

                        @SGaist I just added reusing the same CTranslate Translator, this seems to prevent the memory leak from becoming a problem. Right now I save every Translator that has been used and so I never have to create one more than once. This prevents them from being leaked but ideally I'd like to just save the one that's most recently been used so if someone does a large number of translations without restarting the application they don't have to all be kept in memory. My concern would be that if I did that whatever has been causing this memory leak would also cause the Translator objects to be leaked.

                        Saving the Translators seems to mostly work around the problem but there does seem to be either something wrong with the way I was using PyQt/CTranslate in the example above or a bug in one of them.

                        1 Reply Last reply
                        0

                        15/16

                        31 Oct 2020, 14:44

                        • Login

                        • Login or register to search.
                        15 out of 16
                        • First post
                          15/16
                          Last post
                        0
                        • Categories
                        • Recent
                        • Tags
                        • Popular
                        • Users
                        • Groups
                        • Search
                        • Get Qt Extensions
                        • Unsolved