problem processing standardoutput

Dijkstra

when running my app from terminal i face a problem where the stop function runs before the gpuProcessStandardOutput where i store data and calulcate their average in the stop function any help
the output looks like this
average gpu utilization : -0
average memory used : -0
starting GPU monitor
gpuProcessStarted
gpuProcessStandardOutput
gpu utilization : 25
memory used : 591
gpu utilization : 26
memory used : 591
gpu utilization : 24
memory used : 591
gpu utilization : 24
memory used : 591
gpu utilization : 26
memory used : 591
gpu utilization : 23
memory used : 591
gpu utilization : 24
memory used : 591
gpu utilization : 25
memory used : 591
gpu utilization : 25
memory used : 591
gpu utilization : 23
memory used : 591
gpuProcessFinished
the average should be computed after data is stored not before them

the start function code
code_text

QStringList params;
    if(offline)
        g.offline=true;
    params << "--query-gpu=timestamp,utilization.gpu,memory.used --format=csv ";
    //params << "/home/dijkstra/Downloads/devTools/testing/CPU.py";
    //cpuMonitorProcess.start("python3", params);
    //gpuMonitorProcess.start("nvidia-smi" ,params);
    gpuMonitorProcess.start("nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1");
    QObject::connect(&gpuMonitorProcess , SIGNAL(readyReadStandardOutput()), this, SLOT(gpuProcessStandardOutput()));
    QObject::connect(&gpuMonitorProcess , SIGNAL(finished(int,QProcess::ExitStatus)), this, SLOT(gpuProcessFinished(int, QProcess::ExitStatus)));
    QObject::connect(&gpuMonitorProcess , SIGNAL(started()), this, SLOT(gpuProcessStarted()))

main

QString inputFileNameJSON = argv[1]; //;
        ApplicationEngine myApplicationEngine;
        //myApplicationEngine.checkAndStartServers();
        myApplicationEngine.initPluginOutputListner(inputFileNameJSON);
        /*while(!myApplicationEngine.isProcessCompleted())
        {
            QThread::msleep(100);

        }*/
        myApplicationEngine.parseAccuracyCalculationParams(inputFileNameJSON);
        myApplicationEngine.ComputeAccuracy();
        CPU_monitor cpu;
        cpu.startCPU(true);
        myApplicationEngine.startGPUMonitor(true);
        for(int i=0;i<100;i++){
            QThread::msleep(100);
        }
        myApplicationEngine.stopGPUMonitor(true);
        cpu.stopCPU(true);

jsulm

@Dijkstra Please explain better! What stop function?
Where in your app do you start Qt event loop?

Dijkstra

@jsulm

void ApplicationEngine::stopGPUMonitor(bool offline)
{
    std::cout << "stopping GPU monitor" << std::endl;
    int sum_gpu=0,memory_sum=0;
    for(int i=0;i<g.gpu_utilization_values.size();i++){
        sum_gpu+=g.gpu_utilization_values[i];
        //cout<<g.gpu_utilization_values[i]<<endl;
    }
    for(int i=1;i<g.gpu_utilization_memory_values.size();i++){
        memory_sum+=g.gpu_utilization_memory_values[i];
    }
    int size=g.gpu_utilization_values.size(),memory_size=g.gpu_utilization_memory_values.size();
    memory_size--;
    size--;
    if(!offline){
        emit(sendgpuupdates(QString("%1").arg((float)sum_gpu/size),"average gpu utilization: ",QString("%1").arg((float)memory_sum/memory_size),"average memory usage"));
    }
    g.gpu_utilization_values.clear();
    g.gpu_utilization_memory_values.clear();
    cout<<"average gpu utilization : "<<(float)sum_gpu/size<<endl;
    cout<<"average memory used : " <<(float)memory_sum/memory_size<<endl;

    std::cout << "starting GPU monitor" << std::endl;
         //   cpuMonitorProcess.kill();
    gpuMonitorProcess.terminate();
    gpuMonitorProcess.waitForFinished(-1);
}

this is the stop function where i calculate the average on the data i get from this function

void ApplicationEngine::gpuProcessStandardOutput()
{

    std::cout << "gpuProcessStandardOutput" << std::endl;

    QTextStream rsyncStdoutStream(gpuMonitorProcess.readAllStandardOutput());
    while (true)
      {
          QString line = rsyncStdoutStream.readLine();
          if (line.isNull()) {
              g.finished=true;
              break;
          } else
          {

          QString utilization="",memory_used="";
           //stream << line<<'\n'; //print to file
           for(int i=0;i<line.length();i++){
             if(line[i]!=' ')
                 utilization+=line[i];
             else{
                 while(line[i]!='M'){
                     if(line[i].isDigit())
                         memory_used+=line[i];
                     i++;
                 }
                 break;
             }
           }
           //cout<<memory_used.toStdString()<<endl;

           g.gpu_utilization_memory_values.push_back(memory_used.toInt());
           g.gpu_utilization_values.push_back(utilization.toInt());
           if(!g.offline){
               emit(sendgpuupdates(QString("%1").arg(line),"gpu utilization: ",QString("%1").arg(memory_used),"memory used"));
           }
           //std::cout<<line.toStdString()<<'\n';
           if(utilization.length()!=15)
            std::cout<<"gpu utilization : "<<utilization.toStdString()<<std::endl;
           if(memory_used.length())
            std::cout<<"memory used : "<<memory_used.toStdString()<<std::endl;
          }
      }
}

jsulm

@Dijkstra said in problem processing standardoutput:

for(int i=0;i<100;i++){
QThread::msleep(100);
}

Please do not do such things in event driven frameworks like Qt!
This is not going to work!
Use a QTimer to call stopGPUMonitor...

Dijkstra

@jsulm so i am new can you give me a hint on how to change it to use qtimer ?

jsulm

@Dijkstra I suggest you first learn Qt basics. Also the documentation has examples how to use QTimer: https://doc.qt.io/qt-5/qtimer.html
Here how to use QTimer:

#cpu.startCPU(true);
myApplicationEngine.startGPUMonitor(true);
QTimer::singleShot(10000, [&myApplicationEngine](){ myApplicationEngine.stopGPUMonitor(true); });

And I repeat my question: where in your code do you start Qt event loop?

Dijkstra

@jsulm
starting GPU monitor done
QObject::startTimer: Timers can only be used with threads started with QThread
bye bye now
QProcess: Destroyed while process ("nvidia-smi") is still running.
gpuProcessStarted
gpuProcessFinished

this is the output its stops without printing any data
and i dont have a qt event loop
if u mean the start function this is it
gpuMonitorProcess.start("nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 1");
its in the first code i added in the post

jsulm

@Dijkstra said in problem processing standardoutput:

and i dont have a qt event loop

Then how is this going to work? You need running Qt event loop if you want to use signals/slots.

"QObject::startTimer: Timers can only be used with threads started with QThread" - do you use multi threading?
And please show your current code with QTimer...

"if u mean the start function this is it" - no, that is not what I mean. What I mean belongs to absolute Qt basics:

int main(int argc, char* argv[])
{
    QScopedPointer<QCoreApplication> app(createApplication(argc, argv));

    if (qobject_cast<QApplication *>(app.data())) {
       // start GUI version...
    } else {
       // start non-GUI version...
    }

    return app->exec(); // This starts Qt event loop
}

https://doc.qt.io/qt-5/qapplication.html

Dijkstra

@jsulm

if(argc <=1)
    {
        QApplication a(argc, argv);
        MainWindow w;
        w.show();
        return a.exec();
    }
    else
    {
        QString inputFileNameJSON = argv[1]; //;
        ApplicationEngine myApplicationEngine;
        //myApplicationEngine.checkAndStartServers();
        myApplicationEngine.initPluginOutputListner(inputFileNameJSON);
        /*while(!myApplicationEngine.isProcessCompleted())
        {
            QThread::msleep(100);

        }*/
        myApplicationEngine.parseAccuracyCalculationParams(inputFileNameJSON);
        myApplicationEngine.ComputeAccuracy();
        CPU_monitor cpu;
        //cpu.startCPU(true);
        myApplicationEngine.startGPUMonitor(true);
        /*for(int i=0;i<100;i++){
            QThread::msleep(100);
        }*/
        QTimer::singleShot(10000, [&myApplicationEngine](){ myApplicationEngine.stopGPUMonitor(true); });
        //myApplicationEngine.stopGPUMonitor(true);
        //cpu.stopCPU(true);

        std::cout << "bye bye now" << std::endl;

    }

first condition starts the gui version and the other the non gui one and i am sorry for not knowing this basics w qt

jsulm

@Dijkstra Simply start the event loop also in your second condition like you do in your first. Event loop is not a GUI thing - you can use it in console applications also (and you usually have to if you want to use, for example, signals/slots).

Dijkstra

@jsulm man thanks so much you are amazing it worked is their a way to stop it after it finishing as it doesnt reach the cout<<"bye bye now at the end "<<endl; as i also want to run the cpu and stop it at the same time of stoping the gpu

jsulm

@Dijkstra said in problem processing standardoutput:

is their a way to stop it after it finishing

QTimer::singleShot(10000, [&myApplicationEngine](){ myApplicationEngine.stopGPUMonitor(true); qApp->exit() });

https://doc.qt.io/qt-5/qcoreapplication.html#exit

Dijkstra

@jsulm sorry one last question this is what i did and it works just fine it just doesnt exit the process and keeps hanging in the terminal

QTimer::singleShot(10000, [&myApplicationEngine,&cpu](){
            myApplicationEngine.stopGPUMonitor(true);
            cpu.stopCPU(true);qApp->exit();
            std::cout << "bye bye now" << std::endl;
            qApp->exit(0);
        });

jsulm

@Dijkstra Do you have any other loops somewhere?
Is it asking you to press a button to terminate?

Dijkstra

@jsulm
it doesnt ask for a button to terminate but there is a loop in the function that processes standard output

void ApplicationEngine::gpuProcessStandardOutput()
{

    //std::cout << "gpuProcessStandardOutput" << std::endl;

    QTextStream rsyncStdoutStream(gpuMonitorProcess.readAllStandardOutput());
    while (true)
      {
          QString line = rsyncStdoutStream.readLine();
          if (line.isNull()) {
              g.finished=true;
              break;
          } else
          {

          QString utilization="",memory_used="";
           //stream << line<<'\n'; //print to file
           for(int i=0;i<line.length();i++){
             if(line[i]!=' ')
                 utilization+=line[i];
             else{
                 while(line[i]!='M'){
                     if(line[i].isDigit())
                         memory_used+=line[i];
                     i++;
                 }
                 break;
             }
           }
           //cout<<memory_used.toStdString()<<endl;

           g.gpu_utilization_memory_values.push_back(memory_used.toInt());
           g.gpu_utilization_values.push_back(utilization.toInt());
           if(!g.offline){
               emit(sendgpuupdates(QString("%1").arg(line),"gpu utilization: ",QString("%1").arg(memory_used),"memory used"));
           }
           //std::cout<<line.toStdString()<<'\n';
           if(utilization.length()!=15)
            std::cout<<"gpu utilization : "<<utilization.toStdString()<<std::endl;
           if(memory_used.length())
            std::cout<<"memory used : "<<memory_used.toStdString()<<std::endl;
          }
      }
}

jsulm

@Dijkstra said in problem processing standardoutput:

but there is a loo

Well, make sure you leave that loop

Dijkstra

@jsulm well i checked and all loops does stop but qApp->exit(0) doesnt stop the terminal and i have to press ctrl c to stop it even though their isnt any more output