QtWebEngine uses up all memory and will not give it back!

JonB

Qt 5.12.2. Python/PyQt (though that does not help, it should not be the issue). Tested under Linux, known from user to happen under Windows too. I am in trouble, and I need help from someone who knows their QtWebEngine!

Briefly: I have to create and delete QtWebEngines (for non-interactive use, read below) a large number of times from a loop which cannot call the main event loop. Every instance holds onto its memory allocation --- which is "large" --- until code finally returns to main event loop. I cannot find any way of getting Qt to release the memory being use by QtWebEngine as it proceeds, only when return to main event loop. Result is whole machine runs out of memory + swap space, until it dies/freezes machine, requiring reboot!

In large body of code, QWebEngineView is employed in a QDialog.
Sometimes that dialog is used interactively by user.
But it is also used non-interactively in order to use its ability to print from HTML to PDF file.
Code will do a non-interactive "batch run" of hundreds/thousands of pieces of HTML, exporting to PDF file(s).
During this large amounts of memory will be gobbled by QtWebEngine. Of the order of hundreds of create/delete taking Gigabytes of machine memory. So much so that machine can even run out of all memory and die!
Only a return to top-level, main Qt event loop allows that memory to be recouped. I need something better than that!

I paste below about as minimal an example of code I am using in a test to prove behaviour.

import sys

from PyQt5 import QtCore, QtWidgets
from PyQt5.QtWebEngineWidgets import QWebEngineView


class MyDialog(QtWidgets.QDialog):
    def __init__(self, parent=None):
        super(MyDialog, self).__init__(parent)

        self.wev = QWebEngineView(self)

        self.renderLoop = QtCore.QEventLoop(self)
        self.rendered = False
        self.wev.loadFinished.connect(self.synchronousWebViewLoaded)

        # set some HTML
        html = "<html><body>Hello World</body></html>"
        self.wev.setHtml(html)

        # wait for the HTML to finish rendering asynchronously
        # see synchronousWebViewLoaded() below
        if not self.rendered:
            self.renderLoop.exec()

        # here I would do the printing in real code
        # but it's not necessary to include this to show the memory problem

    def synchronousWebViewLoaded(self, ok: bool):
        self.rendered = True
        # cause the self.renderLoop.exec() above to exit now
        self.renderLoop.quit()


class MyMainWindow(QtWidgets.QMainWindow):
    def __init__(self, parent=None):
        super(MyMainWindow, self).__init__(parent)

        self.btn = QtWidgets.QPushButton("Do Test", self)
        self.btn.clicked.connect(self.doTest)

    def doTest(self):
        print("Started\n")
        # create & delete 500 non-interactive dialog instances
        for i in range(500):
            # create the dialog, it loads the HTML and waits till it has finished rendering
            dlg = MyDialog(self)
            # try desperately to get to delete the dialog/webengine to reclaim memory...
            dlg.deleteLater()
            # next lines do not help from Python
            # del dlg
            # dlg = None
        QtWidgets.QMessageBox.information(self, "Dismiss to return to main event loop", "At this point memory is still in use :(")


if __name__ == '__main__':
    # -*- coding: utf-8 -*-

    app = QtWidgets.QApplication(sys.argv)

    mainWin = MyMainWindow()
    mainWin.show()

    sys.exit(app.exec())

I have tried various flavours of processEvents() in the loop after deleteLater() but none releases the memory in use. Only, only when the code returns to the top-level, main Qt event loop does it get released. Which is too late.

To monitor what is going on, under Linux I used

watch -n 1 free mem
watch -n 1 ps -C QtWebEngineProc
top -o %MEM

There are two areas of memory hogging:

The process itself uses up considerable memory per QtWebEngine
It will run 26 (yes, precisely 26) QtWebEngineProc processes to service the requests, each also taking memory.

Both of these disappear as & when return to Qt top-level event loop, so we know the memory can & should be released. I do not know if this behaviour is QtWebEngine specific.

Anyone kind enough to answer will need to be specific about what to put where to resolve or try out, as I say I have tried a lot of fiddling! Unfortunately, advising to do the whole thing "a different way" (e.g. "do not use QtWebEngineView", "rewrite code so it does not have to do hundreds at a time", etc.) is really not what I am looking for, I need to understand why I can't get it to release its memory as it is now? Can anyone make my deleteLater() release its memory without going back to the top-level Qt event loop??

JonB

To answer my own topic....

From a reply gained via the PyQt mailing list (pyqt@riverbankcomputing.com), it appears the only way to keep memory consumption down is indeed to delete as one goes along. deletelater() does no good for that if one is not hitting the top-level event loop, which can often be the case. From PyQt this must be achieved via sip.delete(widget). This sip.delete() is apparently effectively the equivalent of C++ delete (which, as I said, in itself does not exist in Python). May be of use to any PyQt-ers who need the equivalent.

Armed with this, I can create 1000s of these dialogs with QWebEngineViews, and observe them being deleted and the number of QtWebEngineProcs sticking at 2 as I go along, with the memory being released back to the OS so that it all works OK.

JonB

To answer my own topic....

From a reply gained via the PyQt mailing list (pyqt@riverbankcomputing.com), it appears the only way to keep memory consumption down is indeed to delete as one goes along. deletelater() does no good for that if one is not hitting the top-level event loop, which can often be the case. From PyQt this must be achieved via sip.delete(widget). This sip.delete() is apparently effectively the equivalent of C++ delete (which, as I said, in itself does not exist in Python). May be of use to any PyQt-ers who need the equivalent.

Armed with this, I can create 1000s of these dialogs with QWebEngineViews, and observe them being deleted and the number of QtWebEngineProcs sticking at 2 as I go along, with the memory being released back to the OS so that it all works OK.

Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

QtWebEngine uses up all memory and will not give it back!