Running external python script from QT Application

JonB

You can do all this. startDetached() is the wrong call. Use start(). Do not use waitForFinished(), that is what causes "blocking". You can read output from start()ed process, and as it goes. Look at QProcess::readyRead...() signals, attach slots. There is a also a finished() signal if you need that (instead of the waitForFinished()).

SGaist

Hi,

Having read your Python script: why involve Python at all ? Your script invokes an existing binary with some parameters.
You could directly do that with QProcess and remove a layer of complexity that does not add any benefit.
From a high level view, Popen is equivalent to QProcess.

lukutis222

@SGaist
Yeah I guess you are right, the reason why I chose to run this via Python script is because I plan to run other Python scripts in my program in the future, so if I managed to do everything now, it is going to make it much easier to adjust my application for other Python scripts in the future.

Additionally, I plan to extend this particular Python script to automatically scan for serial ports and choose the right port for flashing. Since I am familliar with Python it is going to be much easier to do in Python than in other executable formats.

lukutis222

@JonB

I have looked into readyRead signal. Please see my code below:


QProcess p; //Global instance of QProcess

void MainWindow::Update_ESP32()
{
    qDebug("Trying to flash esp32 \n");
    QDir dir(QCoreApplication::applicationDirPath());
    QString location = dir.relativeFilePath("./tools/"); // script location
    QStringList params;
    QProcessEnvironment env = QProcessEnvironment::systemEnvironment();
    env.insert("PYTHONPATH", "C:/Users/petrikas.lu/AppData/Local/Programs/Python/Python311/Lib/site-packages");
    env.insert("PYTHONHOME", "C:/Users/petrikas.lu/AppData/Local/Programs/Python/Python311");
    p.setProcessEnvironment(env);
    p.setWorkingDirectory(location);
    connect(&p, &QProcess::errorOccurred, this, &MainWindow::processError);
    connect(&p, &QProcess::finished, this, &MainWindow::flash_finished);
    connect(&p, &QProcess::readyRead, this, &MainWindow::update_percentage);
    params << "C:/Users/petrikas.lu/Desktop/WORK/uCurrent/QT/uCurrent/build-uCurrent-Desktop_Qt_6_4_0_MinGW_64_bit-Debug/tools/flash_esp32.py";
    p.setReadChannel(QProcess::StandardOutput);
    p.start("python", params);
}


void MainWindow::processError(QProcess::ProcessError error){
    qDebug() << "error enum val = " << error << Qt::endl;
}

void MainWindow::flash_finished(){
    qDebug("flash finished \n");
}

void MainWindow::update_percentage(){
        while(p.canReadLine()){
            QString p_stdout = p.readLine();
            qDebug() << p_stdout;
        }

}

I explain the code above:

I create global instance of QProcess. I dont think this is an ideal but I do this so I can access QProcess in my update_percentage slot that is connected to QProcess::readyRead signal.
In my update_percentage method I am checking if there is a line available for reading and read this line.

The results are not what is expected. I am still not getting everything printed line by line. When I run this Python script via QT application it does not print anything as the script is being executed. After the script has successfully executed, it prints everything at once which is exactly what I am trying to avoid:

b'esptool.py v4.1-dev\r\n'
b'Serial port COM3\r\n'
b'Connecting.........\r\n'
b'Chip is ESP32-S3\r\n'
b'Features: WiFi, BLE\r\n'
b'Crystal is 40MHz\r\n'
b'MAC: 68:b6:b3:2d:00:54\r\n'
b'Uploading stub...\r\n'
b'Running stub...\r\n'
b'Stub running...\r\n'
b'Changing baud rate to 460800\r\n'
b'Changed.\r\n'
b'Configuring flash size...\r\n'
b'Auto-detected Flash size: 8MB\r\n'
b'Flash will be erased from 0x00000000 to 0x00055fff...\r\n'
b'Warning: some reserved header fields have non-zero values. This image may be from a newer esptool.py?\r\n'
b'Warning: Image file at 0x0 is protected with a hash checksum, so not changing the flash size setting. Use the --flash_size=keep option instead of --flash_size=8MB in order to remove this warning\r\n'
b'Compressed 351888 bytes to 157257...\r\n'
b'Writing at 0x00000000... (10 %)\r\n'
b'Writing at 0x00011a31... (20 %)\r\n'
b'Writing at 0x0001fd98... (30 %)\r\n'
b'Writing at 0x00026060... (40 %)\r\n'
b'Writing at 0x0002c81a... (50 %)\r\n'
b'Writing at 0x00032a9a... (60 %)\r\n'
b'Writing at 0x0003cc4f... (70 %)\r\n'
b'Writing at 0x0004535c... (80 %)\r\n'
b'Writing at 0x0004b8e5... (90 %)\r\n'
b'Writing at 0x00051cda... (100 %)\r\n'
b'Wrote 351888 bytes (157257 compressed) at 0x00000000 in 4.3 seconds (effective 648.4 kbit/s)...\r\n'
b'Hash of data verified.\r\n'
b'\r\n'
b'Leaving...\r\n'
b'Hard resetting via RTS pin...\r\n'
FLASH_OK

Perhaps you could help me understand what could be the issue?

Pl45m4

@lukutis222 said in Running external python script from QT Application:

Additionally, I plan to extend this particular Python script to automatically scan for serial ports and choose the right port for flashing. Since I am familliar with Python it is going to be much easier to do in Python than in other executable formats.

Why you chose to use C++ then?
Makes no sense to invoke a Python script from C++ every time for additional functionality because you can't do it / don't want to do it in C++.
Then you can better use Qt with Python via PySide or PyQt.

lukutis222

@Pl45m4

This is an older Application built on QT C++ that needs to be extended with the funcionality that I am trying to implement. I am not going to rewrite the whole thing in PyQt just becauese I need to run Python script in my application.

I thought that simply running Python in QProcess is the simplest method to achieve this. If you could suggest a better method I would be happy to hear it.

Pl45m4

@lukutis222 said in Running external python script from QT Application:

I explain the code above:

I create global instance of QProcess. I dont think this is an ideal but I do this so I can access QProcess in my update_percentage slot that is connected to QProcess::readyRead signal.

You should really learn C++ or use Python.
Instead of global, make QProcess a private member of QMainWindow. Then you can also access it within your slot.

In my update_percentage method I am checking if there is a line available for reading and read this line.

The results are not what is expected. I am still not getting everything printed line by line. When I run this Python script via QT application it does not print anything as the script is being executed. After the script has successfully executed, it prints everything at once which is exactly what I am trying to avoid

Probably because you are blocking the signals with your while loop until everything is read and afterwards it's printed all at once.

lukutis222

@Pl45m4 said in Running external python script from QT Application:

You should really learn C++ or use Python.

I am not fully understanding what is your problem. I am literally learning C++ as I am writing code, when I get stuck I sometimes get help here, is that a problem? Unless you expect me to naturally know everything ?

Are you suggesting that if I do not know everything about C++ I should avoid using it? That is garbage logic.

@Pl45m4 said in Running external python script from QT Application:

Instead of global, make QProcess a private member of QMainWindow. Then you can also access it within your slot.

Thanks, that is a good advice.

@Pl45m4 said in Running external python script from QT Application:

Probably because you are blocking the signals with your while loop until everything is read and afterwards it's printed all at once.

Looking into this now. Hopefully get it solved soon :)

lukutis222

I have been further experimenting with readyRead and I think that is where my problems lies. I think I do not fully understand how and when it is expected to trigger. I have read:
https://doc.qt.io/qt-6/qiodevice.html#readyRead

To simply test the readyRead I have written a simply Python script that prints Hello 10 times:

import time

def test_function():
    print("Hello from python script")
    for i in range(10):
        print("Hello \n")
        time.sleep(1)
test_function()

I then run this Python script via QT application and I am monitoring readyRead:

    connect(&update_process, &QProcess::readyRead, this, &MainWindow::update_percentage);

void MainWindow::update_percentage(){
    qDebug("readyRead signal triggered \n");
}

As can be seen from above, I simply want to check how many times my update_percentage method has been triggered. The result - only once at the end of the Python program execution.

Trying to flash esp32 
readyRead signal triggered 
flash finished

I am currently reading and trying to understand how and why it behaves this way as I expected this method to trigger more than once since multiple blocks of serial data were received.

JonB

@lukutis222 said in Running external python script from QT Application:

The results are not what is expected. I am still not getting everything printed line by line. When I run this Python script via QT application it does not print anything as the script is being executed. After the script has successfully executed, it prints everything at once which is exactly what I am trying to avoid:

Your Qt code looks correct to me. There are a couple of possibilities for why you say you get all the output in "one blob" instead of progressively:

The Python application buffers its print() output at its side, and it only gets sent back to the Qt invoking process at the end/after a lot of output. Look up Python documentation for this (run as python -u? Set PYTHONUNBUFFERED environment variable? Flush stdout after each print()?)
From Python you run the sub-program via Popen(), via iter(pipe.readline). Do you know that states the readlines return in real time as soon as each line is available, not just at the end/after a while?
Similar to first point, do you know that ESP32_Programmer_path process sends its output down the pipe immediately, no buffering at its side?

As always, simplify while you develop/discover. Remove the Python layer, and its subprocess. Try something like

process.start("cmd", { "/c", "dir /s \\" });

That produces a lot of output over time, See how that arrives to your Qt program, to check you do get that as it goes. Then, even if you wish to run Python scripts longterm, try running the ESP32_Programmer_path command directly from your Qt program instead of via the Python wrapper.

As @Pl45m4 said, it is slightly strange that you want to write so much as Python script sub-programs from your Qt C++ program. You can of course do that if you wish, or if you want to learn. If you have an existing C++ program and you do not want to rewrite in Python/PyQt/PySide you might consider either running Python directly inside your C++ (not subprocess), there is a module to allow this, or incorporating the Python scripts' functionality directly into the C++ for a "smoother" experience (e.g. for the example you have I agree with @SGaist that it would actually be much simpler to just run the ESP32_Programmer_path command directly from C++/Qt). But up to you, and indeed maybe the simplest is these subprocesses.

lukutis222

@JonB
Yep.. It totally was the issue with the stdout not being flushed after print in python script.

def log_subprocess_output(pipe):
    for line in iter(pipe.readline, b''): # b'\n'-separated lines
        print(line)
        sys.stdout.flush()

What caused issues was simply because when I run the script via the command line, it prints out the data nicely (not in one blob) hence I expected it to work the same way in QT. It turned out that its not exactly working like that.

I have learned how to run the Python script via the QT application. As you have suggested, for this particular use case I will try to use ESP32_Programmer_path directly and see how it goes.

Thanks!

JonB

When you run the script from a terminal its stdout is to terminal; when you run it from Qt program or other its stdout is to pipe of some kind. Python/Windows may do different buffering for these, hence different behaviour. Don't forget you can also run the/all Python script(s) with python -u or PYTHONUNBUFFERED environment variable if you don't want to have to insert code after every print() to flush stdout.

Pl45m4

@lukutis222

Since you want to read stdout from your scripts, you could also try to use the QProcess::readyReadStandardOutput signal instead of readyRead.

https://doc.qt.io/qt-6/qprocess.html#readyReadStandardOutput

Then in the connected function, call QProcess::readAllStandardOutput.

https://doc.qt.io/qt-6/qprocess.html#readAllStandardOutput

Does it make a difference?