Solved QtWebEngine uses up all memory and will not give it back!
-
Qt 5.12.2. Python/PyQt (though that does not help, it should not be the issue). Tested under Linux, known from user to happen under Windows too. I am in trouble, and I need help from someone who knows their QtWebEngine!
Briefly: I have to create and delete QtWebEngines (for non-interactive use, read below) a large number of times from a loop which cannot call the main event loop. Every instance holds onto its memory allocation --- which is "large" --- until code finally returns to main event loop. I cannot find any way of getting Qt to release the memory being use by QtWebEngine as it proceeds, only when return to main event loop. Result is whole machine runs out of memory + swap space, until it dies/freezes machine, requiring reboot!
-
In large body of code,
QWebEngineView
is employed in aQDialog
. -
Sometimes that dialog is used interactively by user.
-
But it is also used non-interactively in order to use its ability to print from HTML to PDF file.
-
Code will do a non-interactive "batch run" of hundreds/thousands of pieces of HTML, exporting to PDF file(s).
-
During this large amounts of memory will be gobbled by QtWebEngine. Of the order of hundreds of create/delete taking Gigabytes of machine memory. So much so that machine can even run out of all memory and die!
-
Only a return to top-level, main Qt event loop allows that memory to be recouped. I need something better than that!
I paste below about as minimal an example of code I am using in a test to prove behaviour.
import sys from PyQt5 import QtCore, QtWidgets from PyQt5.QtWebEngineWidgets import QWebEngineView class MyDialog(QtWidgets.QDialog): def __init__(self, parent=None): super(MyDialog, self).__init__(parent) self.wev = QWebEngineView(self) self.renderLoop = QtCore.QEventLoop(self) self.rendered = False self.wev.loadFinished.connect(self.synchronousWebViewLoaded) # set some HTML html = "<html><body>Hello World</body></html>" self.wev.setHtml(html) # wait for the HTML to finish rendering asynchronously # see synchronousWebViewLoaded() below if not self.rendered: self.renderLoop.exec() # here I would do the printing in real code # but it's not necessary to include this to show the memory problem def synchronousWebViewLoaded(self, ok: bool): self.rendered = True # cause the self.renderLoop.exec() above to exit now self.renderLoop.quit() class MyMainWindow(QtWidgets.QMainWindow): def __init__(self, parent=None): super(MyMainWindow, self).__init__(parent) self.btn = QtWidgets.QPushButton("Do Test", self) self.btn.clicked.connect(self.doTest) def doTest(self): print("Started\n") # create & delete 500 non-interactive dialog instances for i in range(500): # create the dialog, it loads the HTML and waits till it has finished rendering dlg = MyDialog(self) # try desperately to get to delete the dialog/webengine to reclaim memory... dlg.deleteLater() # next lines do not help from Python # del dlg # dlg = None QtWidgets.QMessageBox.information(self, "Dismiss to return to main event loop", "At this point memory is still in use :(") if __name__ == '__main__': # -*- coding: utf-8 -*- app = QtWidgets.QApplication(sys.argv) mainWin = MyMainWindow() mainWin.show() sys.exit(app.exec())
I have tried various flavours of
processEvents()
in the loop afterdeleteLater()
but none releases the memory in use. Only, only when the code returns to the top-level, main Qt event loop does it get released. Which is too late.To monitor what is going on, under Linux I used
watch -n 1 free mem watch -n 1 ps -C QtWebEngineProc top -o %MEM
There are two areas of memory hogging:
- The process itself uses up considerable memory per QtWebEngine
- It will run 26 (yes, precisely 26)
QtWebEngineProc
processes to service the requests, each also taking memory.
Both of these disappear as & when return to Qt top-level event loop, so we know the memory can & should be released. I do not know if this behaviour is QtWebEngine specific.
Anyone kind enough to answer will need to be specific about what to put where to resolve or try out, as I say I have tried a lot of fiddling! Unfortunately, advising to do the whole thing "a different way" (e.g. "do not use QtWebEngineView", "rewrite code so it does not have to do hundreds at a time", etc.) is really not what I am looking for, I need to understand why I can't get it to release its memory as it is now? Can anyone make my
deleteLater()
release its memory without going back to the top-level Qt event loop?? -
-
To answer my own topic....
From a reply gained via the PyQt mailing list (pyqt@riverbankcomputing.com), it appears the only way to keep memory consumption down is indeed to delete as one goes along.
deletelater()
does no good for that if one is not hitting the top-level event loop, which can often be the case. From PyQt this must be achieved viasip.delete(widget)
. Thissip.delete()
is apparently effectively the equivalent of C++delete
(which, as I said, in itself does not exist in Python). May be of use to any PyQt-ers who need the equivalent.Armed with this, I can create 1000s of these dialogs with
QWebEngineView
s, and observe them being deleted and the number ofQtWebEngineProc
s sticking at 2 as I go along, with the memory being released back to the OS so that it all works OK. -
To answer my own topic....
From a reply gained via the PyQt mailing list (pyqt@riverbankcomputing.com), it appears the only way to keep memory consumption down is indeed to delete as one goes along.
deletelater()
does no good for that if one is not hitting the top-level event loop, which can often be the case. From PyQt this must be achieved viasip.delete(widget)
. Thissip.delete()
is apparently effectively the equivalent of C++delete
(which, as I said, in itself does not exist in Python). May be of use to any PyQt-ers who need the equivalent.Armed with this, I can create 1000s of these dialogs with
QWebEngineView
s, and observe them being deleted and the number ofQtWebEngineProc
s sticking at 2 as I go along, with the memory being released back to the OS so that it all works OK.