Headless GUI testing with Qt?

[patrickkidd]

Has anyone had success setting up Qt unit tests for QtWidgets/Qml that do not require real native windows to receive active focus? This prevents unittests from being run in the background and also slows them down considerably.

I am currently calling QWidget:::activateWindow() and waiting for windows to become active, for example using QTest::qWaitForWindowActive. I also have a lot of shims that use QTest::mouseClick, QTest::keyPress, etc. to throw everything possible at various element types to have it predictably gain and lose focus. I also have ported these techniques to Qml with the same level of success. However, these are slow, not 100% reliable, and often - but not always - require foreground status of the test process.

I am on macOS. And though it probably doesn't directly affect my question, I am running PyQt5 & pytest with the Qt plugin (3.2.2).

How has others solved this problem? Thanks!

[SGaist]

Hi,

Are you using the offscreen backend ?

[patrickkidd]

@SGaist If you mean -platform offscreen then no, because the docs say that it is only supported on X11.

[SGaist]

Can you show an example of such a test failing ?

I am using pytest-qt on both macOS and Linux with the offscreen backend but with widgets. There might be some different constraints for your situation.

[patrickkidd]

@SGaist The common case is where an input widget like TextInput doesn’t receive focus unless the app is in the foreground so onTextChanged is never called. My own code therefore fails on an assertion when the item fails to receive focus. I’m not sure that pasting an example of that will add much to the conversation? This does not fail in all cases, but I haven’t been able to isolate the cases it does fail on.

Are you saying that you successfully run tests using the offscreen plugin on macOS?

[SGaist]

Yes, that's what I am saying. But as I wrote, it's for a 100% widget application.

An example of failing unit test would help use find what we are doing differently that may be explored to get your test more reliable.

[patrickkidd]

@SGaist Shoot, it is always so hard to pull out an example when it is an integration test. I'll see what I can do but chances are low.

I am curious, do you just pass ['-platform, 'offscreen'] to your QApplication constructor in a fixture? I tried this for some of my QGraphicsScene|Item tests and it just froze.

[SGaist]

No, I am using the QT_QPA_PLATFORM environnement variable both in the CI and development machine.

[patrickkidd]

@SGaist said in Headless GUI testing with Qt?:

QT_QPA_PLATFORM

Good call with the env var. It appears that the following test passes with the default platform but fails at assert w.hasFocus() for QT_QPA_PLATFORM=offscreen.

def test_active_window(qtbot):
    w = QLineEdit()
    w.show()
    qtbot.keyClicks(w, 'here we are')
    assert w.hasFocus()
    assert w.text() == 'here we are'

Thoughts?

[patrickkidd]

It also looks like the following test passes when QT_QPA_PLATFORM=offscreen, but fails when run together in a test suite with the previous QtWidgets-only test. The QtWidgets-only test fails in every case for QT_QPA_PLATFORM=offscreen. When QT_QPA_PLATFORM is blank, both tests pass when run both separately and when run together as a test suite. Hmmm...

def test_qml_active_item(qtbot):

    class QQW(QQuickWidget):
        @pyqtSlot(str)
        def textChanged(self, text):
            print(text)
            self._text = text
    qml = QQW()
    qml.rootContext().setContextProperty('qml', qml)
    qml.setSource(QUrl.fromLocalFile('./stuff.qml'))
    qtbot.mouseClick(qml, Qt.LeftButton, Qt.NoModifier)
    qtbot.keyClicks(qml, 'here we are')
    assert qml.rootObject().property('input').hasActiveFocus() == True
    assert qml.rootObject().property('text') == 'here we are'
    assert qml._text == 'here we are'

stuff.qml:

import QtQuick 2.12

Item {
    property var input: textInput
    property string text: textInput.text

    TextInput {
	id: textInput
        onTextChanged: qml.textChanged(text)
    }
}

[SGaist]

@patrickkidd said in Headless GUI testing with Qt?:

def test_active_window(qtbot):
w = QLineEdit()
w.show()
qtbot.keyClicks(w, 'here we are')
assert w.hasFocus()
assert w.text() == 'here we are'

if you add qtbot.waitUntil(lambda: w.hasFocus()) before your asserts, then it works as expected with both tests in a row.

[patrickkidd]

@SGaist said in Headless GUI testing with Qt?:

if you add qtbot.waitUntil(lambda: w.hasFocus()) before your asserts, then it works as expected with both tests in a row.

Now that is very interesting. It works here too. I wonder why waiting for the first QtWidgets test to gain focus fixes the qml widget test when run as a suite?

[SGaist]

That's an pretty good question but I don't currently have any idea, sorry.

It might be some sort of internal state that is not updated properly.

Did you check by any chance if you have the same behaviour with PySide2 ?

[patrickkidd]

@SGaist I’ve never used PySide2 but that would make sense. This has been a productive thread so far, I’d say. A lot of my tests work as is except my c++ engine was getting stuck waiting on a macOS queue that didn’t dispatch, and the same focus errors I originally complained about here. I have a feeling I’ll discover some problems fixing those and will leave the thread open for that.