Running external python script from QT Application

[lukutis222]

In my project debug folder, I have created a folder tools . This folder contains an a Python script that I want to run via QT Application. I tried to do the following in my QT Application:

void MainWindow::Update_ESP32()
{

    QDir dir(QCoreApplication::applicationDirPath());
    QString location = dir.relativeFilePath("./tools/");

    QString  command("python");
    QStringList params = QStringList() << "flash_esp32.py";

    QProcess *process = new QProcess();
    process->startDetached(command, params, location);
    process->waitForFinished();
    process->close();

}

The error I get:

Python path configuration:
  PYTHONHOME = 'C:\
Qt\Tools\m
ingw
1120
_64\
bin\..
\opt'
  PYTHONPATH = (not set)
  program name = 'p
ython'

  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = 'C
:\Qt\T
oo
ls\m
ingw11
20_64\
opt\
Lib'

  sys._base_executable = 'C:\\Users\\petrikas.lu\\AppData\\Local\\Programs\\Python\\Python311\\python.exe'
  sys.base_prefix = 'C:\\Qt\\Tools\\mingw1120_64\\bin\\..\\opt'

  sys.base_exec_prefix = 'C:\\Qt\\Tools\\mingw1120_64\\bin\\..\\opt'
  sys.platlibdir = 'DLLs'
  sys.executable = 'C:\\Users\\petrikas.lu\\AppData\\Local\\Programs\\Python\\Python311\\python.exe'
  sys.prefix = 'C:\\Qt\\Tools\\mingw1120_64\\bin\\..\\opt'
  sys.exec_prefix = 'C:\\Qt\\Tools\\mingw1120_64\\bin\\..\\opt'
  sys.path = [
    'C:\\Users\\petrikas.lu\\AppData\\Local\\Programs\\Python\\Python311\\python311.zip',
    'C:\\Qt\\Tools\\mingw1120_64\\opt\\Lib',
    'C:\\Qt\\Tools\\mingw1120_64\\opt\\DLLs',
  ]
Fatal Python error: init_fs_encoding: 
failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError
: No module named 'encodings'

Current thread 0x000046c8
 (most recent call first):
  <no Python frame>

I have tried to find a solution to this issue but cannot seem to understand what is the best solution. I have also read that setting environment variables (PYTHONHOME and PYTHONPATH) is not good idea.
https://discuss.python.org/t/fatal-python-error-init-fs-encoding-failed-to-get-the-python-codec-of-the-filesystem-encoding/3173

1. I have Python and Python 3 installed on my PC and can run the script via the CMD without any issues. I want to test and run this python script via QT application.

2. The next step would be to use the local Python environment for my application. This way users would be able to run my code even if they do not have Python installed on their machine

I would appreciate if someone could clarify and suggest a method to solve the issue and allow QT applications to run python scripts

[JonB]

@lukutis222 said in Running external python script from QT Application:

The results are not what is expected. I am still not getting everything printed line by line. When I run this Python script via QT application it does not print anything as the script is being executed. After the script has successfully executed, it prints everything at once which is exactly what I am trying to avoid:

Your Qt code looks correct to me. There are a couple of possibilities for why you say you get all the output in "one blob" instead of progressively:

• The Python application buffers its print() output at its side, and it only gets sent back to the Qt invoking process at the end/after a lot of output. Look up Python documentation for this (run as python -u? Set PYTHONUNBUFFERED environment variable? Flush stdout after each print()?)
• From Python you run the sub-program via Popen(), via iter(pipe.readline). Do you know that states the readlines return in real time as soon as each line is available, not just at the end/after a while?
• Similar to first point, do you know that ESP32_Programmer_path process sends its output down the pipe immediately, no buffering at its side?

As always, simplify while you develop/discover. Remove the Python layer, and its subprocess. Try something like

process.start("cmd", { "/c", "dir /s \\" });

That produces a lot of output over time, See how that arrives to your Qt program, to check you do get that as it goes. Then, even if you wish to run Python scripts longterm, try running the ESP32_Programmer_path command directly from your Qt program instead of via the Python wrapper.

As @Pl45m4 said, it is slightly strange that you want to write so much as Python script sub-programs from your Qt C++ program. You can of course do that if you wish, or if you want to learn. If you have an existing C++ program and you do not want to rewrite in Python/PyQt/PySide you might consider either running Python directly inside your C++ (not subprocess), there is a module to allow this, or incorporating the Python scripts' functionality directly into the C++ for a "smoother" experience (e.g. for the example you have I agree with @SGaist that it would actually be much simpler to just run the ESP32_Programmer_path command directly from C++/Qt). But up to you, and indeed maybe the simplest is these subprocesses.

[Axel Spoerl]

Just dump the environment on the command line with SET and check which variables are relevant for running a Python script on the command line.
Then set the process environment accordingly.

[lukutis222]

@Axel-Spoerl
Thanks for reply. I have made some significant progress.

My Python is installed at:
C:\Users\my_user\AppData\Local\Programs\Python\Python311

In QT, I do the following:

void MainWindow::Update_ESP32()
{
    QStringList params;
    QProcess p;
    connect(&p, &QProcess::errorOccurred, this, &MainWindow::processError);
    connect(&p, &QProcess::finished, this, &MainWindow::flash_finished);

    QProcessEnvironment env = QProcessEnvironment::systemEnvironment();
    env.insert("PYTHONPATH", "C:/Users/my_user/AppData/Local/Programs/Python/Python311/Lib/site-packages");
    env.insert("PYTHONHOME", "C:/Users/my_user/AppData/Local/Programs/Python/Python311");
    p.setProcessEnvironment(env);

    params << "C:/Users/my_user/Desktop/WORK/uCurrent/QT/uCurrent/build-uCurrent-Desktop_Qt_6_4_0_MinGW_64_bit-Debug/tools/flash_esp32.py";
    p.start("python", params);
    p.waitForFinished(-1);
    qDebug() << "finished";
    QString p_stdout = p.readAll();
    qDebug() << p_stdout;
    QString p_stderr = p.readAllStandardError();
    if(!p_stderr.isEmpty())
       qDebug()<<"Python error:"<<p_stderr;
}

void MainWindow::processError(QProcess::ProcessError error){
    qDebug() << "error enum val = " << error << Qt::endl;
}

void MainWindow::flash_finished(){
    qDebug("flash finished \n");
}

The error message when I try to run the application:

flash finished 
finished
""
Python error: "Traceback (most recent call last):\r\n  File \"C:\\Users\\my_user\\Desktop\\WORK\\uCurrent\\QT\\uCurrent\\build-uCurrent-Desktop_Qt_6_4_0_MinGW_64_bit-Debug\\tools\\flash_esp32.py\", line 29, in <module>\r\n    Flash_ESP32()\r\n  File \"C:\\Users\\my_user\\Desktop\\WORK\\uCurrent\\QT\\uCurrent\\build-uCurrent-Desktop_Qt_6_4_0_MinGW_64_bit-Debug\\tools\\flash_esp32.py\", line 16, in Flash_ESP32\r\n    process = Popen([ESP32_Programmer_path, '-p', 'COM3', '-b','460800','--before','default_reset','--after','hard_reset','--chip', 'esp32s3','write_flash','--flash_mode','dio','--flash_size','detect','--flash_freq','80m','0x0',HEX_PATH], stdout=PIPE, stderr=STDOUT)\r\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"C:\\Users\\my_user\\AppData\\Local\\Programs\\Py�޵G�

As you can see, I have made some progress and it actually tried to execute the python script, although I am not sure what is the issue since the error message is not clear nor informative.

I can run the script via cmdline without any issues which confirms that the Python script itself works fine:

C:\Users\my_user\Desktop\WORK\uCurrent\QT\uCurrent\build-uCurrent-Desktop_Qt_6_4_0_MinGW_64_bit-Debug\tools>python flash_esp32.py
b'esptool.py v4.1-dev\r\n'
b'Serial port COM3\r\n'
b'Connecting.........\r\n'
b'Chip is ESP32-S3\r\n'
b'Features: WiFi, BLE\r\n'
b'Crystal is 40MHz\r\n'
b'MAC: 68:b6:b3:2d:00:54\r\n'
b'Uploading stub...\r\n'
b'Running stub...\r\n'
b'Stub running...\r\n'
b'Changing baud rate to 460800\r\n'
b'Changed.\r\n'
b'Configuring flash size...\r\n'
b'Auto-detected Flash size: 8MB\r\n'
b'Flash will be erased from 0x00000000 to 0x00055fff...\r\n'
b'Warning: some reserved header fields have non-zero values. This image may be from a newer esptool.py?\r\n'
b'Warning: Image file at 0x0 is protected with a hash checksum, so not changing the flash size setting. Use the --flash_size=keep option instead of --flash_size=8MB in order to remove this warning\r\n'
b'Compressed 351888 bytes to 157257...\r\n'
b'Writing at 0x00000000... (10 %)\r\n'
b'Writing at 0x00011a31... (20 %)\r\n'
b'Writing at 0x0001fd98... (30 %)\r\n'
b'Writing at 0x00026060... (40 %)\r\n'
b'Writing at 0x0002c81a... (50 %)\r\n'
b'Writing at 0x00032a9a... (60 %)\r\n'
b'Writing at 0x0003cc4f... (70 %)\r\n'
b'Writing at 0x0004535c... (80 %)\r\n'
b'Writing at 0x0004b8e5... (90 %)\r\n'
b'Writing at 0x00051cda... (100 %)\r\n'
b'Wrote 351888 bytes (157257 compressed) at 0x00000000 in 4.3 seconds (effective 650.8 kbit/s)...\r\n'
b'Hash of data verified.\r\n'
b'\r\n'
b'Leaving...\r\n'
b'Hard resetting via RTS pin...\r\n'
FLASH_OK

The contents of my Python script:

from subprocess import Popen, PIPE, STDOUT
import os
import re

ESP32_Programmer_path = 'esptool.exe'
HEX_PATH = 'uCurrent-1.1.0_full.bin'

def log_subprocess_output(pipe):
    for line in iter(pipe.readline, b''): # b'\n'-separated lines
        print(line)
        
def Flash_ESP32():
    process = Popen([ESP32_Programmer_path, '-p', 'COM3', '-b','460800','--before','default_reset','--after','hard_reset','--chip', 'esp32s3','write_flash','--flash_mode','dio','--flash_size','detect','--flash_freq','80m','0x0',HEX_PATH], stdout=PIPE, stderr=STDOUT)
    with process.stdout:
        log_subprocess_output(process.stdout)
    exitcode = process.wait() # 0 means success
    if(exitcode == 0):
        print("FLASH_OK")
        return 1;
    else:
        print("FLASH_FAIL")
        return 0;
        


Flash_ESP32()

As you can see the Python script calls the esptool.exe with the required arguments. esptool.exe sits in the same directory where the Python script is.

[Axel Spoerl]

OK, so you run python code as an external script, that itself calls an external script.
In that case, you need to set a the script location as a working directory.

[lukutis222]

@Axel-Spoerl

Yep. That solved the issue. Thanks!

If anyone is interested, this is the code:

void MainWindow::Update_ESP32()
{
    qDebug("Trying to flash esp32 \n");
    QDir dir(QCoreApplication::applicationDirPath());
    QString location = dir.relativeFilePath("./tools/"); // script location
    QStringList params;
    QProcess p;

    QProcessEnvironment env = QProcessEnvironment::systemEnvironment();
    env.insert("PYTHONPATH", "C:/Users/my_user/AppData/Local/Programs/Python/Python311/Lib/site-packages");
    env.insert("PYTHONHOME", "C:/Users/my_user/AppData/Local/Programs/Python/Python311");
    p.setProcessEnvironment(env);
    p.setWorkingDirectory(location);

    connect(&p, &QProcess::errorOccurred, this, &MainWindow::processError);
    connect(&p, &QProcess::finished, this, &MainWindow::flash_finished);

    params << "C:/Users/my_user/Desktop/WORK/uCurrent/QT/uCurrent/build-uCurrent-Desktop_Qt_6_4_0_MinGW_64_bit-Debug/tools/flash_esp32.py";

    p.start("python", params);
    p.waitForFinished(-1);
    qDebug() << "finished";
    QString p_stdout = p.readAll();
    qDebug() << p_stdout;
    QString p_stderr = p.readAllStandardError();
    if(!p_stderr.isEmpty())
       qDebug()<<"Python error:"<<p_stderr;
}


void MainWindow::processError(QProcess::ProcessError error){
    qDebug() << "error enum val = " << error << Qt::endl;
}

void MainWindow::flash_finished(){
    qDebug("flash finished \n");
}

Just one more clarification though - I am not able to start this in detached process mode. If I replace:

    p.start("python", params);
    p.waitForFinished(-1);
    qDebug() << "finished";
    QString p_stdout = p.readAll();
    qDebug() << p_stdout;
    QString p_stderr = p.readAllStandardError();
    if(!p_stderr.isEmpty())
       qDebug()<<"Python error:"<<p_stderr;

With:

     p.startDetached("python", params);
     p.waitForFinished();
     p.close();

I get the following error:

Trying to flash esp32 
Python path configuration:
  PYTHONHOME = 'C
:\Qt\Tools
\min
gw11
20_6
4\bin\
..\opt
'
  PYTHONPATH = (not set)

  program name = 'p
ytho
n'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = 'C
:\Qt
\Too
ls\min
gw11
20_6
4\op
t\Li
b'
  sys._base_executable = 
'C:\\Users\\my_user\\AppData\\Local\\Programs\\Python\\Python311\\python.exe'
  sys.base_prefix = 'C:\\Qt\\Tools\\mingw1120_64\\bin\\..\\opt'

  sys.base_exec_prefix = 'C:\\Qt\\Tools\\mingw1120_64\\bin\\..\\opt'
  sys.platlibdir = 'DLLs'

  sys.executable = 'C:\\Users\\my_user\\AppData\\Local\\Programs\\Python\\Python311\\python.exe'
  sys.prefix = 'C:\\Qt\\Tools\\mingw1120_64\\bin\\..\\opt'
  sys.exec_prefix = 'C:\\Qt\\Tools\\mingw1120_64\\bin\\..\\opt'
  sys.path = [
    'C:\\Users\\my_user\\AppData\\Local\\Programs\\Python\\Python311\\python311.zip',
    'C:\\Qt\\Tools\\mingw1120_64\\opt\\Lib',
    'C:\\Qt\\Tools\\mingw1120_64\\opt\\DLLs',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError
: No module named 'encodings'

Current thread 0x00004f9c (most recent call first):
  <no Python frame>

Is that related to the environment variables that I set? Is that not correct?

    env.insert("PYTHONPATH", "C:/Users/my_user/AppData/Local/Programs/Python/Python311/Lib/site-packages");
    env.insert("PYTHONHOME", "C:/Users/my_user/AppData/Local/Programs/Python/Python311");
    p.setProcessEnvironment(env);

The next step is to figure out how can embed the python locally to my application. I want to be able to generate windows executable out of my application and I want it to be able to run the python script even if a person does not have Python installed on his PC.

[JonB]

@lukutis222

• You can't waitForFinished() if you startDetached().
• The overload of startDetached() you use is static. Everything you did with p and env has no effect.

[lukutis222]

@JonB

Ah I see.. The reason why starting process in blocking mode is not ideal for me is because the Python script is printing data periodically as it is flashing the external MCU

b'esptool.py v4.1-dev\r\n'
b'Serial port COM3\r\n'
b'Connecting.........\r\n'
b'Chip is ESP32-S3\r\n'
b'Features: WiFi, BLE\r\n'
b'Crystal is 40MHz\r\n'
b'MAC: 68:b6:b3:2d:00:54\r\n'
b'Uploading stub...\r\n'
b'Running stub...\r\n'
b'Stub running...\r\n'
b'Changing baud rate to 460800\r\n'
b'Changed.\r\n'
b'Configuring flash size...\r\n'
b'Auto-detected Flash size: 8MB\r\n'
b'Flash will be erased from 0x00000000 to 0x00055fff...\r\n'
b'Warning: some reserved header fields have non-zero values. This image may be from a newer esptool.py?\r\n'
b'Warning: Image file at 0x0 is protected with a hash checksum, so not changing the flash size setting. Use the --flash_size=keep option instead of --flash_size=8MB in order to remove this warning\r\n'
b'Compressed 351888 bytes to 157257...\r\n'
b'Writing at 0x00000000... (10 %)\r\n'
b'Writing at 0x00011a31... (20 %)\r\n'
b'Writing at 0x0001fd98... (30 %)\r\n'
b'Writing at 0x00026060... (40 %)\r\n'
b'Writing at 0x0002c81a... (50 %)\r\n'
b'Writing at 0x00032a9a... (60 %)\r\n'
b'Writing at 0x0003cc4f... (70 %)\r\n'
b'Writing at 0x0004535c... (80 %)\r\n'
b'Writing at 0x0004b8e5... (90 %)\r\n'
b'Writing at 0x00051cda... (100 %)\r\n'
b'Wrote 351888 bytes (157257 compressed) at 0x00000000 in 4.3 seconds (effective 650.4 kbit/s)...\r\n'
b'Hash of data verified.\r\n'
b'\r\n'
b'Leaving...\r\n'
b'Hard resetting via RTS pin...\r\n'
FLASH_OK

I was hoping to read the data while the process is running and display the FW update progress (percentage) in my QT Application as the update is going through. I was hoping to achieve this by starting detached process.

Since my process is blocking till its complete, I cannot read the data that Python script prints as it is running. I only get the output after the Python script completes

[JonB]

You can do all this. startDetached() is the wrong call. Use start(). Do not use waitForFinished(), that is what causes "blocking". You can read output from start()ed process, and as it goes. Look at QProcess::readyRead...() signals, attach slots. There is a also a finished() signal if you need that (instead of the waitForFinished()).

[SGaist]

Hi,

Having read your Python script: why involve Python at all ? Your script invokes an existing binary with some parameters.
You could directly do that with QProcess and remove a layer of complexity that does not add any benefit.
From a high level view, Popen is equivalent to QProcess.

[lukutis222]

@SGaist
Yeah I guess you are right, the reason why I chose to run this via Python script is because I plan to run other Python scripts in my program in the future, so if I managed to do everything now, it is going to make it much easier to adjust my application for other Python scripts in the future.

Additionally, I plan to extend this particular Python script to automatically scan for serial ports and choose the right port for flashing. Since I am familliar with Python it is going to be much easier to do in Python than in other executable formats.

[lukutis222]

@JonB

I have looked into readyRead signal. Please see my code below:

QProcess p; //Global instance of QProcess

void MainWindow::Update_ESP32()
{
    qDebug("Trying to flash esp32 \n");
    QDir dir(QCoreApplication::applicationDirPath());
    QString location = dir.relativeFilePath("./tools/"); // script location
    QStringList params;
    QProcessEnvironment env = QProcessEnvironment::systemEnvironment();
    env.insert("PYTHONPATH", "C:/Users/petrikas.lu/AppData/Local/Programs/Python/Python311/Lib/site-packages");
    env.insert("PYTHONHOME", "C:/Users/petrikas.lu/AppData/Local/Programs/Python/Python311");
    p.setProcessEnvironment(env);
    p.setWorkingDirectory(location);
    connect(&p, &QProcess::errorOccurred, this, &MainWindow::processError);
    connect(&p, &QProcess::finished, this, &MainWindow::flash_finished);
    connect(&p, &QProcess::readyRead, this, &MainWindow::update_percentage);
    params << "C:/Users/petrikas.lu/Desktop/WORK/uCurrent/QT/uCurrent/build-uCurrent-Desktop_Qt_6_4_0_MinGW_64_bit-Debug/tools/flash_esp32.py";
    p.setReadChannel(QProcess::StandardOutput);
    p.start("python", params);
}


void MainWindow::processError(QProcess::ProcessError error){
    qDebug() << "error enum val = " << error << Qt::endl;
}

void MainWindow::flash_finished(){
    qDebug("flash finished \n");
}

void MainWindow::update_percentage(){
        while(p.canReadLine()){
            QString p_stdout = p.readLine();
            qDebug() << p_stdout;
        }

}

I explain the code above:

1. I create global instance of QProcess. I dont think this is an ideal but I do this so I can access QProcess in my update_percentage slot that is connected to QProcess::readyRead signal.

2. In my update_percentage method I am checking if there is a line available for reading and read this line.

The results are not what is expected. I am still not getting everything printed line by line. When I run this Python script via QT application it does not print anything as the script is being executed. After the script has successfully executed, it prints everything at once which is exactly what I am trying to avoid:

b'esptool.py v4.1-dev\r\n'
b'Serial port COM3\r\n'
b'Connecting.........\r\n'
b'Chip is ESP32-S3\r\n'
b'Features: WiFi, BLE\r\n'
b'Crystal is 40MHz\r\n'
b'MAC: 68:b6:b3:2d:00:54\r\n'
b'Uploading stub...\r\n'
b'Running stub...\r\n'
b'Stub running...\r\n'
b'Changing baud rate to 460800\r\n'
b'Changed.\r\n'
b'Configuring flash size...\r\n'
b'Auto-detected Flash size: 8MB\r\n'
b'Flash will be erased from 0x00000000 to 0x00055fff...\r\n'
b'Warning: some reserved header fields have non-zero values. This image may be from a newer esptool.py?\r\n'
b'Warning: Image file at 0x0 is protected with a hash checksum, so not changing the flash size setting. Use the --flash_size=keep option instead of --flash_size=8MB in order to remove this warning\r\n'
b'Compressed 351888 bytes to 157257...\r\n'
b'Writing at 0x00000000... (10 %)\r\n'
b'Writing at 0x00011a31... (20 %)\r\n'
b'Writing at 0x0001fd98... (30 %)\r\n'
b'Writing at 0x00026060... (40 %)\r\n'
b'Writing at 0x0002c81a... (50 %)\r\n'
b'Writing at 0x00032a9a... (60 %)\r\n'
b'Writing at 0x0003cc4f... (70 %)\r\n'
b'Writing at 0x0004535c... (80 %)\r\n'
b'Writing at 0x0004b8e5... (90 %)\r\n'
b'Writing at 0x00051cda... (100 %)\r\n'
b'Wrote 351888 bytes (157257 compressed) at 0x00000000 in 4.3 seconds (effective 648.4 kbit/s)...\r\n'
b'Hash of data verified.\r\n'
b'\r\n'
b'Leaving...\r\n'
b'Hard resetting via RTS pin...\r\n'
FLASH_OK

Perhaps you could help me understand what could be the issue?

[Pl45m4]

@lukutis222 said in Running external python script from QT Application:

Additionally, I plan to extend this particular Python script to automatically scan for serial ports and choose the right port for flashing. Since I am familliar with Python it is going to be much easier to do in Python than in other executable formats.

Why you chose to use C++ then?
Makes no sense to invoke a Python script from C++ every time for additional functionality because you can't do it / don't want to do it in C++.
Then you can better use Qt with Python via PySide or PyQt.

[lukutis222]

@Pl45m4

This is an older Application built on QT C++ that needs to be extended with the funcionality that I am trying to implement. I am not going to rewrite the whole thing in PyQt just becauese I need to run Python script in my application.

I thought that simply running Python in QProcess is the simplest method to achieve this. If you could suggest a better method I would be happy to hear it.

[Pl45m4]

@lukutis222 said in Running external python script from QT Application:

I explain the code above:

I create global instance of QProcess. I dont think this is an ideal but I do this so I can access QProcess in my update_percentage slot that is connected to QProcess::readyRead signal.

You should really learn C++ or use Python.
Instead of global, make QProcess a private member of QMainWindow. Then you can also access it within your slot.

In my update_percentage method I am checking if there is a line available for reading and read this line.

The results are not what is expected. I am still not getting everything printed line by line. When I run this Python script via QT application it does not print anything as the script is being executed. After the script has successfully executed, it prints everything at once which is exactly what I am trying to avoid

Probably because you are blocking the signals with your while loop until everything is read and afterwards it's printed all at once.

[lukutis222]

@Pl45m4 said in Running external python script from QT Application:

You should really learn C++ or use Python.

I am not fully understanding what is your problem. I am literally learning C++ as I am writing code, when I get stuck I sometimes get help here, is that a problem? Unless you expect me to naturally know everything ?

Are you suggesting that if I do not know everything about C++ I should avoid using it? That is garbage logic.

@Pl45m4 said in Running external python script from QT Application:

Instead of global, make QProcess a private member of QMainWindow. Then you can also access it within your slot.

Thanks, that is a good advice.

@Pl45m4 said in Running external python script from QT Application:

Probably because you are blocking the signals with your while loop until everything is read and afterwards it's printed all at once.

Looking into this now. Hopefully get it solved soon :)

[lukutis222]

I have been further experimenting with readyRead and I think that is where my problems lies. I think I do not fully understand how and when it is expected to trigger. I have read:
https://doc.qt.io/qt-6/qiodevice.html#readyRead

To simply test the readyRead I have written a simply Python script that prints Hello 10 times:

import time

def test_function():
    print("Hello from python script")
    for i in range(10):
        print("Hello \n")
        time.sleep(1)
test_function()

I then run this Python script via QT application and I am monitoring readyRead:

    connect(&update_process, &QProcess::readyRead, this, &MainWindow::update_percentage);

void MainWindow::update_percentage(){
    qDebug("readyRead signal triggered \n");
}

As can be seen from above, I simply want to check how many times my update_percentage method has been triggered. The result - only once at the end of the Python program execution.

Trying to flash esp32 
readyRead signal triggered 
flash finished 

I am currently reading and trying to understand how and why it behaves this way as I expected this method to trigger more than once since multiple blocks of serial data were received.

[JonB]

@lukutis222 said in Running external python script from QT Application:

The results are not what is expected. I am still not getting everything printed line by line. When I run this Python script via QT application it does not print anything as the script is being executed. After the script has successfully executed, it prints everything at once which is exactly what I am trying to avoid:

Your Qt code looks correct to me. There are a couple of possibilities for why you say you get all the output in "one blob" instead of progressively:

• The Python application buffers its print() output at its side, and it only gets sent back to the Qt invoking process at the end/after a lot of output. Look up Python documentation for this (run as python -u? Set PYTHONUNBUFFERED environment variable? Flush stdout after each print()?)
• From Python you run the sub-program via Popen(), via iter(pipe.readline). Do you know that states the readlines return in real time as soon as each line is available, not just at the end/after a while?
• Similar to first point, do you know that ESP32_Programmer_path process sends its output down the pipe immediately, no buffering at its side?

As always, simplify while you develop/discover. Remove the Python layer, and its subprocess. Try something like

process.start("cmd", { "/c", "dir /s \\" });

That produces a lot of output over time, See how that arrives to your Qt program, to check you do get that as it goes. Then, even if you wish to run Python scripts longterm, try running the ESP32_Programmer_path command directly from your Qt program instead of via the Python wrapper.

As @Pl45m4 said, it is slightly strange that you want to write so much as Python script sub-programs from your Qt C++ program. You can of course do that if you wish, or if you want to learn. If you have an existing C++ program and you do not want to rewrite in Python/PyQt/PySide you might consider either running Python directly inside your C++ (not subprocess), there is a module to allow this, or incorporating the Python scripts' functionality directly into the C++ for a "smoother" experience (e.g. for the example you have I agree with @SGaist that it would actually be much simpler to just run the ESP32_Programmer_path command directly from C++/Qt). But up to you, and indeed maybe the simplest is these subprocesses.

[lukutis222]

@JonB
Yep.. It totally was the issue with the stdout not being flushed after print in python script.

def log_subprocess_output(pipe):
    for line in iter(pipe.readline, b''): # b'\n'-separated lines
        print(line)
        sys.stdout.flush() 

What caused issues was simply because when I run the script via the command line, it prints out the data nicely (not in one blob) hence I expected it to work the same way in QT. It turned out that its not exactly working like that.

I have learned how to run the Python script via the QT application. As you have suggested, for this particular use case I will try to use ESP32_Programmer_path directly and see how it goes.

Thanks!

[JonB]

When you run the script from a terminal its stdout is to terminal; when you run it from Qt program or other its stdout is to pipe of some kind. Python/Windows may do different buffering for these, hence different behaviour. Don't forget you can also run the/all Python script(s) with python -u or PYTHONUNBUFFERED environment variable if you don't want to have to insert code after every print() to flush stdout.

[Pl45m4]

@lukutis222

Since you want to read stdout from your scripts, you could also try to use the QProcess::readyReadStandardOutput signal instead of readyRead.

• https://doc.qt.io/qt-6/qprocess.html#readyReadStandardOutput

Then in the connected function, call QProcess::readAllStandardOutput.

• https://doc.qt.io/qt-6/qprocess.html#readAllStandardOutput

Does it make a difference?