How to Implement Asynchronous Operations for PySide Called from Qt C++

Teni · wrote on 16 Oct 2024, 08:02

Title: How to Implement Asynchronous Operations for PySide Called from Qt C++

Qt C++ Version: 6.5.3/6.8.0
PySide Version: 6.5.3/6.8.0

I'm currently trying to call PySide code from Qt C++. Below is a snippet of my code:

QWidget *plugin_widget = new QWidget();
uintptr_t plugin_widget_ptr = reinterpret_cast<uintptr_t>(plugin_widget);

PyObject *pArgs = Py_BuildValue("(K)", plugin_widget_ptr);
PyGILState_STATE gstate;
gstate = PyGILState_Ensure(); 

PyObject *pWidget = PyObject_CallObject(pFuncCreateWidget, pArgs);

PyGILState_Release(gstate);

In this part, I'm passing a QWidget created in Qt C++ to Python.

def create_plugin_widget(widget_ptr: QWidget) -> QWidget:
    if type(widget_ptr) is QWidget:
        widget = widget_ptr
    else:
        widget = wrapInstance(ctypes.c_void_p(widget_ptr).value, QWidget)
        widget.setStyleSheet("QLabel { color: white; }")

    widget.plugin_manager = PluginWidgetManager(widget)

In Python, I'm using wrapInstance to convert the Qt C++ object into a Python QWidget object and manipulating it within the PluginWidgetManager class.

self.widget = parent_widget
self.layout = QVBoxLayout(self.widget)

self.init_data = {
    'Password_A': "FFFFFFFFFFFF",
    'Password_B': "FFFFFFFFFFFF",
    'Shop_ID': "1"
}

self.serial_thread = SerialThread(self.init_data)
self.serial_thread.signals.detect_signal.connect(self.handle_detect_signal)
self.serial_thread.signals.init_signal.connect(self.handle_init_signal)
self.serial_thread.signals.write_signal.connect(self.handle_write_signal)
self.serial_thread.start()

I've implemented serial operations using QThread. However, while the QThread works normally in Python, when called from C++, the main page is fine, but the QThread gets stuck in the background. It doesn't stop but rather accumulates. If the page remains inactive for a long time, a sudden request can trigger all accumulated thread operations, causing the program to crash.

I’ve tried using asyncio, QThread, and threading with no success. I would appreciate any help on how to resolve this issue.

Teni · wrote on 19 Oct 2024, 12:09

I successfully resolved this issue. The root cause was that in Qt C++, it is necessary to define PyThreadState *state = PyEval_SaveThread(); to enable threading capabilities in the called code. Additionally, the definition of PyThreadState *state = PyEval_SaveThread(); must be present inside int main, regardless of whether you call the Python code from other functions or, like me, from within the called DLL.

int main(int argc, char *argv[]) {
    QApplication app(argc, argv);
    config_path = QCoreApplication::applicationDirPath() + "/config.ini";
    load_app_config();
    if (show_loginwindow() == 0)
    {
        return 0;
    }
    PyThreadState *state = PyEval_SaveThread();
    return app.exec();
}

SGaist · 17 Oct 2024, 01:59

Hi and welcome to devnet,

Can you explain the goal of your implementation ? It's pretty convoluted to try to pass a QWidget object from some C++ part to some unrelated Python part of the same application.

Teni · S SGaist 16 Oct 2024, 19:46

@SGaist I would like to establish a plugin distribution framework.

This framework will involve building a plugin manager in C++, supporting applications developed in various languages such as C++ and Python.

In the plugin manager, I determine the interpreter by locating the Python installation in the system path and place some necessary Python libraries in the program directory.

Using methods provided by Python.h, I pass one of the QWidget instances from the page implemented by the plugin manager to Python. As you can see, it is converted to uintptr_t type before being passed. After passing, it can be restored to a QWidget object that can be manipulated by PySide using Shiboken's wrapInstance function.

For setLayout, addWidget, and even general signals, everything can be bound and triggered without any issues.

However, higher-level features like QThread and asyncio cannot be triggered at all. Using a QTimer for periodic processing can trigger once during initialization, but afterward, unless any widget is interacted with to activate the plugin program, it will remain in a stalled state.

This means that while PySide plugins can implement pages and basic logic, they are completely unable to handle long-running tasks.

Teni · S SGaist 16 Oct 2024, 19:46

@SGaist

Python：

from PySide6.QtWidgets import QWidget, QVBoxLayout, QLabel
from PySide6.QtCore import QThread, QObject, Signal
from shiboken6 import wrapInstance
import ctypes
import asyncio
class SerialThread(QObject):
    def __init__(self, update_signal: Signal):
        super().__init__()
        self.update_signal = update_signal
        self.counter = 0

    async def start(self):
        while True:
            self.counter += 1
            self.update_signal.emit(str(self.counter))
            await asyncio.sleep(1)

class AsyncWorker(QThread):
    update_signal = Signal(str)

    def __init__(self):
        super().__init__()

    def run(self):
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)
        loop.run_until_complete(self.run_async())

    async def run_async(self):
        serial_thread = SerialThread(self.update_signal)
        await serial_thread.start()

class PluginWidgetManager:
    def __init__(self, parent_widget: QWidget):
        self.widget = parent_widget
        self.layout = QVBoxLayout(self.widget)

        self.main_widget()
        self.worker = AsyncWorker()
        self.worker.update_signal.connect(self.update_label)
        self.worker.start()

    def main_widget(self):
        self.statusbar_widget = QWidget()
        self.statusbar_layout = QVBoxLayout(self.statusbar_widget)
        self.layout.addWidget(self.statusbar_widget)

        self.label = QLabel("0")
        self.statusbar_layout.addWidget(self.label)

    def update_label(self, value: str):
        self.label.setText(value)  

def create_plugin_widget(widget_ptr: QWidget) -> QWidget:
    if type(widget_ptr) is QWidget:
        widget = widget_ptr
    else:
        widget = wrapInstance(ctypes.c_void_p(widget_ptr).value, QWidget)
        widget.setStyleSheet("QLabel { color: white; }")

    widget.plugin_manager = PluginWidgetManager(widget)

def get_secret_key() -> str:
    return "1234567890"

def get_plugin_name() -> str:
    return "Demo"

if __name__ == "__main__":
    import sys
    from PySide6.QtWidgets import QApplication
    app = QApplication(sys.argv)
    widget = QWidget()
    widget.resize(720, 480)
    create_plugin_widget(widget)
    widget.show()
    sys.exit(app.exec())

CPP：

QDir plugins_dir = app_dir;
plugins_dir.cd("plugins");
QFileInfoList sub_dirs = plugins_dir.entryInfoList(QDir::Dirs | QDir::NoDotAndDotDot); // Get subdirectories
QStringList dll_plugin_files;
QStringList pyd_plugin_files;

foreach (QFileInfo sub_dir, sub_dirs) {
    QDir dir(sub_dir.absoluteFilePath()); // Enter each subdirectory
    QStringList dll_file_names = dir.entryList(QStringList() << "*.dll", QDir::Files); // Find .dll files in the subdirectory
    QStringList pyd_file_names = dir.entryList(QStringList() << "*.pyd", QDir::Files); // Find .pyd files in the subdirectory
    foreach (QString dll_file, dll_file_names) {
        dll_plugin_files << dir.absoluteFilePath(dll_file); // Get the absolute path of the file
    }
    foreach (QString pyd_file, pyd_file_names) {
        pyd_plugin_files << dir.absoluteFilePath(pyd_file); // Get the absolute path of the file
    }
}

foreach (QString plugin_file, dll_plugin_files)
{
    library.setFileName(plugin_file);
    auto get_plugin_name = (GetPluginNameFunc)library.resolve("get_plugin_name");
    QString plugin_name = *get_plugin_name();

    auto get_plugin_widget = (GetPluginWidgetFunc)library.resolve("get_plugin_widget");
    QWidget *plugin_widget = get_plugin_widget();

    QPushButton* button = new QPushButton(plugin_name);
    button->setFixedHeight(40);
    pluginlist_layout->addWidget(button);
    pluginview_layout->addWidget(plugin_widget);
    QObject::connect(button, &QPushButton::clicked, this, [this, pluginview_layout, plugin_widget, plugin_name]() {
        // Switch to the corresponding widget
        pluginview_layout->setCurrentWidget(plugin_widget);
        // Update the displayed plugin name in pluginname_label
        this->pluginname_label->setText(plugin_name);
    });
}

foreach (QString plugin_file, pyd_plugin_files)
{
    PyObject* pDict = get_python_plugin_Func(plugin_file);
    if (!pDict) {
        qInfo() << "Failed to get Python plugin function for" << plugin_file;
        continue; // Skip this plugin
    }

    PyObject *pFuncGetPluginName = PyDict_GetItemString(pDict, "get_plugin_name");
    if (!pFuncGetPluginName) {
        qInfo() << "Function get_plugin_name not found for" << plugin_file;
        continue; // Skip this plugin
    }

    PyObject *pResult = PyObject_CallObject(pFuncGetPluginName, NULL);
    QString plugin_name = QString::fromUtf8(PyUnicode_AsUTF8(pResult));

    PyObject *pFuncCreateWidget = PyDict_GetItemString(pDict, "create_plugin_widget");
    if (!pFuncCreateWidget) {
        qInfo() << "Function create_plugin_widget not found for" << plugin_file;
        continue; // Skip this plugin
    }

    QWidget *plugin_widget = new QWidget();
    uintptr_t plugin_widget_ptr = reinterpret_cast<uintptr_t>(plugin_widget);

    PyObject *pArgs = Py_BuildValue("(K)", plugin_widget_ptr);
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure(); // Acquire GIL

    PyObject *pWidget = PyObject_CallObject(pFuncCreateWidget, pArgs);

    // Release GIL
    PyGILState_Release(gstate);

    if (PyErr_Occurred()) {
        // Print error message
        PyErr_Print();
    }

    QPushButton* button = new QPushButton(plugin_name);
    button->setFixedHeight(40);
    pluginlist_layout->addWidget(button);
    pluginview_layout->addWidget(plugin_widget);
    QObject::connect(button, &QPushButton::clicked, this, [this, pluginview_layout, plugin_widget, plugin_name]() {
        pluginview_layout->setCurrentWidget(plugin_widget);
        this->pluginname_label->setText(plugin_name);
    });

    Py_XDECREF(pArgs);
    Py_XDECREF(pResult);
    Py_XDECREF(pWidget);
    Py_XDECREF(pModule);
    Py_XDECREF(pDict);
}

// Dynamically find the Python executable
QString findPythonExecutable() {
    QProcess process;
    process.start("where", QStringList() << "python");

    if (!process.waitForFinished()) {
        qWarning() << "Unable to execute where command";
        return QString();
    }

    QString output = process.readAllStandardOutput();
    QStringList paths = output.split('\n'); // Split into lines
    paths.removeAll(QString()); // Remove empty lines

    // Filter out false Python paths
    QString validPythonPath;
    for (const QString &path : paths) {
        QString trimmedPath = path.trimmed(); // Remove leading and trailing spaces and newline characters
        if (trimmedPath.contains("Microsoft") || trimmedPath.contains("WindowsApps")) {
            continue; // Skip false paths
        }
        validPythonPath = trimmedPath; // Find a valid path
        break; // Exit loop after finding the first valid path
    }

    if (validPythonPath.isEmpty()) {
        qWarning() << "No valid Python executable found";
    } else {
        qInfo() << "Found valid Python executable:" << validPythonPath;
    }

    return validPythonPath;
}

// Used in init_python function
void Mainwindow_logic::init_python() {
    QString pythonExecutable = findPythonExecutable();
    if (pythonExecutable.isEmpty()) {
        qFatal("No Python executable found");
    }

    // Get the installation directory of Python
    QFileInfo pythonInfo(pythonExecutable);
    QString pythonHome = pythonInfo.absolutePath(); // This will be the directory of intelpython3
    QString app_dir = QFileInfo(QCoreApplication::applicationDirPath()).absolutePath() + "/..";
    QString app_sitepackages_path = app_dir + "/site-packages";

    // Set PYTHONHOME and PYTHONPATH based on the Python executable path
    Py_SetPythonHome((wchar_t*)pythonHome.utf16()); // Set PYTHONHOME

    // Initialize Python
    Py_Initialize();

    // Dynamically add site-packages path
    QString sitePackagesPath = pythonHome + "/Lib/site-packages"; // Assume Python is installed in Lib/site-packages
    qInfo() << app_sitepackages_path;
    qInfo() << sitePackagesPath;

    PyRun_SimpleString("import sys");
    PyRun_SimpleString(("sys.path.append(r\"" + sitePackagesPath + "\")").toStdString().c_str());
    PyRun_SimpleString(("sys.path.append(r\"" + app_sitepackages_path + "\")").toStdString().c_str());
}

PyObject* Mainwindow_logic::get_python_plugin_Func(QString pyd_path) {
    // Check if the file extension is `.pyd`
    if (!pyd_path.endsWith(".pyd", Qt::CaseInsensitive)) {
        return nullptr;  // Return nullptr if it's not a .pyd file
    }

    QFileInfo fileInfo(pyd_path);
    QString moduleName = fileInfo.baseName();  // Get the module name without the extension

    pModule = nullptr;
    pDict = nullptr;

    // Add the plugin directory to Python's sys.path
    QString dirPath = fileInfo.absolutePath();
    PyObject* sysPath = PySys_GetObject("path");
    PyObject* path = PyUnicode_DecodeFSDefault(dirPath.toUtf8().constData());
    PyList_Append(sysPath, path);
    Py_DECREF(path);

    // Load the .pyd module
    pModule = PyImport_ImportModule(moduleName.toUtf8().constData());

    pDict = PyModule_GetDict(pModule);

    return pDict;  // Return the dictionary
}

Teni · wrote on 19 Oct 2024, 12:09

I successfully resolved this issue. The root cause was that in Qt C++, it is necessary to define PyThreadState *state = PyEval_SaveThread(); to enable threading capabilities in the called code. Additionally, the definition of PyThreadState *state = PyEval_SaveThread(); must be present inside int main, regardless of whether you call the Python code from other functions or, like me, from within the called DLL.

int main(int argc, char *argv[]) {
    QApplication app(argc, argv);
    config_path = QCoreApplication::applicationDirPath() + "/config.ini";
    load_app_config();
    if (show_loginwindow() == 0)
    {
        return 0;
    }
    PyThreadState *state = PyEval_SaveThread();
    return app.exec();
}

SGaist · wrote on 19 Oct 2024, 14:26

Nice ! Glad you found out and thanks for sharing.

That said, I would suggest to properly clean that state object.