How to automate user tasks in a Desktop app developed in Qt?

SunnyMax

Hello,
I would like to know if and how it is possible to automate repetitive user tasks in a business application (on OSX) that had been developed with Qt.

I am familiar with GUI scripting using Python etc. but this is not what I want to do.
Sending click commands using xy-coordinates and working with image detection is the least reliable way to automate repetitive tasks.

Automating application testing neither is what I intend to do.

What I prefer to achieve is scripting the interactions using the UI elements' properties.
That's why initially I started using AppleScript and various UI Inspector/Accessibility Inspector tools. They finally revealed classNames indicating that the business App must have been developed in Qt.

While I can easily edit text fields with Applescript, other UI elements such as button and date picker do not respond to AppleScript commands. A dropdown like view with autosuggestions when typing into the corresponding text field is another example, where I would like to navigate through the items and pick the one matching my criteria defined in the automation routine.

Since this is my first contact with Qt, though cross reading and searching the Qt documentation, product descriptions as well as the forum, I couldn't find a hint.

Could you please give me some advice where I could find what is necessary to know to automate interaction with the buttons and other UI elements?

Many thanks.

SGaist

Hi and welcome to devnet,

If I understand you correctly, you are just a user of the application without link to its developers ?

SunnyMax

@SGaist yes, exactly, I am just a user of this application. I have no link to the developers. The application is as is.

SGaist

Then maybe through the accessibility API but I can't guarantee that it will work the way you need it.

SunnyMax

After countless hours of research and try and error, here is my outcome for those interested into the same topic:

Let's first define "automation":
Hereby I mean that program code (a script) emulates user actions with the application.
The objective is to free the user from repetitive, boring tasks. This could be for example, data exports on a scheduled basis which are then further analyzed.

What I am not talking about is:

Test automation.
automation of industrial processes.

With this out of the way, I can summarize:

Web applications have an automation API based upon the W3C specification of web driver
Windows applications have an automation API called UIA3. It supports Win32, Windows Form, WPF.

A Qt application running on Windows cannot benefit from Window's automation API. The underlying libraries of course cannot reference and interact with QML/QQuick objects.

If you have a Qt-programmed application that you did not program yourself, there is no way to emulate user interactions!

There is no API that allows you to address the GUI elements programmatically. The Accessibility API requires that the developers would have invested their time to provide means to connect with accessibility applications. This is very often not the case for obvious reasons.

In the QT world the rule applies : No access to the source code, no way to automate !

Squish and Ranorex are commercial tools made for test automation. In theory they could be converted to emulate user interaction with the 3rd party application.

However, for non-commercial automation projects the following reasons speak against this only possible workaround:

the considerable license costs of Squish and Ranorex
extreme overhead installing these tools
necessity to have a separate package for each Qt version, i.e. if you would like to automate an application A coded in qt 6.2 you need to install another package than for a qt application coded in version 6.4. You get it.