Can't get value from command chaining ...

darongyi

Hello! I made QProcess class to work this command.

curl "http://dbpedia.org/data/Haeinsa.json" | perl dbpediaPlaceData.txt 'Haeinsa'

I made class like this.

Problem : One process keep running and I can't toss QByteArray data to perlProcess.
My goal is getting JSON value from perlProcess.

#include "processmanager.h"
#include <QDir>
#include <QDebug>
#include <QUrl>
#include <QStandardPaths>
#include <QRegularExpression>

ProcessManager::ProcessManager(int max_processes, QObject *parent) :
      QObject(parent),
      _num_process(0)
      ,_max_processes(max_processes)
       ,curlProcess(new QProcess(this))
      ,perlProcess(new QProcess(this))
      ,dbpediaPlaceFile("")
      ,mKeyword("")
      ,byteArray()
{

    _timer = new QTimer();
    _timer->setInterval(100);

    connect(_timer, &QTimer::timeout, this, QOverload<>::of(&ProcessManager::start_process));

    createPath();
    // simulates a kill process signal from the network
    //QTimer::singleShot(8000, [this](){kill_process(_sacrifice, "Sacrifical Process");});
}


void ProcessManager::createPath()
{

   QString basicPath = QStandardPaths::writableLocation(QStandardPaths::DownloadLocation);
   dbpediaPlaceFile = QString("%1%2%3").arg(basicPath, QDir::separator(),"dbpediaPlaceData.txt");

   loadDirectory(dbpediaPlaceFile);

}

void ProcessManager::loadDirectory(const QString& path)
{
    QDir dir(path);

    if(!dir.exists())
      dir.mkpath(path);
}

void ProcessManager::setKeyword(const QString &keyword)
{
    mKeyword = keyword;
}

void ProcessManager::start_process(QString name)
{

    QUrl baseUrl = QUrl(QString("http://dbpedia.org/data/%1.json").arg(mKeyword));
    qDebug() << baseUrl;
    if(_num_process == 0)
    {
       _processes.append(curlProcess);
       setup_connections(curlProcess,name);
       curlProcess->setStandardOutputProcess(perlProcess);
    }
    curlProcess->start("curl", {"-s", baseUrl.toString()});

    _num_process++;
  // _timer->stop();
}

void ProcessManager::start_perl(QString name)
{
     _processes.append(perlProcess);
    setup_connections( perlProcess,name);
    perlProcess->start("perl", { dbpediaPlaceFile, mKeyword});

    qDebug() << byteArray;
    perlProcess->write(byteArray);
    perlProcess->closeWriteChannel();

    _timer->stop();
}

void ProcessManager::setup_connections(QProcess *process, QString process_name)
{
    // TODO: Connect the `finished` and `errorOccured` signals up
    connect(process, QOverload<int, QProcess::ExitStatus>::of(&QProcess::finished),
            [process, process_name,this](int exit_code, QProcess::ExitStatus status)
    {

      this->process_finished(exit_code, status, process,process_name);
    });

    connect(process, &QProcess::errorOccurred, [process, process_name,this](QProcess::ProcessError error)
    {
        this->error_handler(error, process,process_name);
    });


    // NOTE: Unused process signals:
    //       started, stateChanged, readyReadStandard[Error/Output]

}

void ProcessManager::process_finished(int exit_code, QProcess::ExitStatus status, QProcess* process,
                                       QString process_name)
{
    if (status == QProcess::NormalExit)
    {
       // qDebug() << "normal exit for process: " << process_name;

        if(process_name.startsWith("curl"))
       {

         QString curlOut = process->readAllStandardOutput();
         qDebug() << "OUTPUT" << curlOut;

         byteArray.append(curlOut);

         QByteArray byteKeyword(mKeyword.toLatin1());

          if(byteArray.indexOf(byteKeyword) != -1)
         {

           connect(_timer, &QTimer::timeout, this, QOverload<>::of(&ProcessManager::start_perlProcess));

           //kill_process(process, "curl");
           _processes.removeOne(process);
           resume();
          }

        }
       else
       {

         _processes.removeOne(process);
       }

    }
    else if (status == QProcess::CrashExit)
    {
       // qWarning() << "Process crashed! Process: " << process_name;
        qDebug() << "Exit code: " << exit_code;
       //process->start(process_name);
    }
}

void ProcessManager::kill_process(QProcess* process, QString name)
{
    qDebug() << "Killed process: " << name;

#ifdef W_OS_WIN
    // Console applications on Windows that do not run an event loop,
    // or whose event loop does not handle the WM_CLOSE message, can only be terminated by calling kill
    process->kill();
#else
    // Nice version
    process->terminate();
    // process->kill();
#endif

    // Not nice version

    return;
}

void ProcessManager::error_handler(QProcess::ProcessError error, QProcess *process,
                                   QString process_name)
{
    qDebug() << "Error!";

  
}

void ProcessManager::start()
{
    qDebug() << "Starting!";
    _timer->start();
   // _mainProcess->start();
}

void ProcessManager::resume()
{
    _timer->start();
}

void ProcessManager::start_process()
{
    start_process(QString("curl"));
}

void ProcessManager::start_perlProcess()
{
    start_perl(QString("perl"));
}

JonB

@darongyi
Terminating processes/collecting exit codes can be problematic in pipe-chained command sequences.

Why do you make it so complicated for yourself? Do yourself a favour, don't create separate processes, why don't you send that command line as a whole to one bash QProcess and let it deal with the pipes & exit codes, and intermediate "byte arrays"...?

darongyi

@JonB sorry to make comlicated class..

JonB

@darongyi
It's OK, but you just do not want to do it your way.
Like said, can't you just use the whole line as a command to the shell?

darongyi

yes
I don't know how can access..

JonB

@darongyi
Sorry, "access" what?

darongyi

@JonB Question change
https://forum.qt.io/topic/93716/i-try-to-use-qprocess-startdetached-funtion

JonB

@darongyi
I have made a post there. I believe you are going down totally the wrong route trying to use startDetached(), and you would have been better persevering here.

JonB

@darongyi
You have about 4 options here.

Stick with your original 2 separate QProcesss (use start() for both, no startDetached()). Now that I have found http://doc.qt.io/qt-5/qprocess.html#setStandardOutputProcess

Pipes the standard output stream of this process to the destination process' standard input.

This actually looks dead-easy. I can't see why you shouldn't just use this way.

Get the OS "shell" to execute the whole of curl "http://dbpedia.org/data/Haeinsa.json" | perl dbpediaPlaceData.txt 'Haeinsa' as a single command for you. More flexible for potential future, maybe. Bit tricky with quoting, especially if you want this to work cross-platform.
Are you in charge of the Perl script yourself? Since you're running Perl you could just do the curl command from within the Perl script, pulling its output in directly.
In your other post I think you were starting to go down the road of first running the curl and having your Qt app store all its output in a string, and then sending that as input to the perl. Avoiding piping. Not good if the output from the curl is large.

All in all, what about giving #1 a go? It should be pretty straightforward, now that I know about QProcess::setStandardOutputProcess() I'd do it that way if I had just your example to implement.

P.S.
You say your Perl command is

perl dbpediaPlaceData.txt 'Haeinsa'

You're really sure about that, and you've tested it, right? Because dbpediaPlaceData.txt seems like a strange name for a Perl script file to me....

SGaist

Hi,

In addition to what @JonB wrote, what exactly does that script do with the JSON data ?

darongyi

@JonB sorry ask same question.
I change simple code using QProcess and fix command.
Thank you for the tip.

curl "http://dbpedia.org/data/Haeinsa.json" | perl dbpediaPlaceData.txt 'Haeinsa'

I made simple code and fix command.
Text file is on github.

ProcessManager::ProcessManager(QObject *parent) :
      QObject(parent),
      _num_process(0)
      ,_max_processes(0)
      ,curlProcess(new QProcess(this))

      ,dbpediaPlaceFile("")
      ,mKeyword("")

{

    _timer = new QTimer();
    _timer->setInterval(100);

    connect(_timer, &QTimer::timeout, this, QOverload<>::of(&ProcessManager::start_process));
    createPath();
    // simulates a kill process signal from the network
    //QTimer::singleShot(8000, [this](){kill_process(_sacrifice, "Sacrifical Process");});
}



void ProcessManager::createPath()
{

   QString basicPath = QStandardPaths::writableLocation(QStandardPaths::DownloadLocation);
   dbpediaPlaceFile = QString("%1%2%3").arg(basicPath, QDir::separator(),"dbpediaPlaceData.txt");

}

void ProcessManager::loadDirectory(const QString& path)
{
    QDir dir(path);

    if(!dir.exists())
      dir.mkpath(path);
}

void ProcessManager::setKeyword(const QString &keyword)
{
    mKeyword = keyword;
}

void ProcessManager::start_process(QString name)
{

    QString cmd;
    QStringList args;

    QByteArray byteerr;
    QByteArray byteout;

    cmd  = QString("curl \"http://dbpedia.org/data/%1.json\""
                    " | perl %2 '%3'").arg(mKeyword).arg(dbpediaPlaceFile).arg(mKeyword);
    args << "-c" << cmd;

    curlProcess->start("/bin/sh",args);
     // Give the child process some time to start

    curlProcess->waitForStarted();
    if(curlProcess->waitForFinished())
    {
        byteerr += curlProcess->readAllStandardError();
        byteout += curlProcess->readAllStandardOutput();

        qDebug() << byteerr;
        qDebug() << byteout;

        emit putLog(byteout,Qt::black);

    }
   _timer->stop();
   curlProcess->close();

}

JonB

@darongyi
In principle it looks OK, in that you have chosen to do the command via /bin/sh and let it handle the piping. That was my option #2. Though I have to say: once we discovered QProcess::setStandardOutputProcess() I don't know why you didn't adopt option #1, but that's up to you.

I don't know what your "// Give the child process some time to start" & "_timer->stop();" are about. For full flexibility you may find that instead of using the two waitFor...() calls you could switch over to using the signal-driven calls. You might want to add some error checking of the QProcess calls to your code --- you never know when a process might fail....

darongyi

I called class on dialog..
I tried to use setstandardoutputprocess.
It didnt’t bring data.. Can u give examples?
I want to improve better..

JonB

@darongyi
I've never used QProcess::setStandardOutputProcess()! But if I did I'd expect it to look exactly like the example in the manual page http://doc.qt.io/qt-5/qprocess.html#setStandardOutputProcess:

The following shell command:

command1 | command2

Can be accomplished with QProcess with the following code:

QProcess process1;
QProcess process2;

process1.setStandardOutputProcess(&process2);

process1.start("command1");
process2.start("command2");

Obviously your #1 is the curl and #2 is the perl. That bit has done the 2 processes connected by the pipe. Then picking up the output from the perl --- if there is any in your case --- should be same as your code readAllStandardOutput/Error() but obviously we want the output from the right-hand perlProcess instead of the left-hand curlProcess. And you will need perlProcess->waitForFinished() instead of curlProcess->waitForFinished().

Having said the above, if you can't get it working I'd stick with what you've got now and just tidy up on any result checking & error handling.

darongyi

I change code. I can't get data.
So, json data is huge. i put code waitforfinished() .

void ProcessManager::start_process(QString name)
{
  QProcess *process1, *process2;

  process1 = new QProcess(this);
  process2 = new QProcess(this);

  QString cmd1, cmd2;
  QStringList args;
  QByteArray Size;

  cmd1  = QString("http://dbpedia.org/data/%1.json").arg(mKeyword);
  cmd2 = QString("perl %1 '%2'").arg(dbpediaPlaceFile).arg(mKeyword);

//  args << "-c" << cmd;

  qDebug() << cmd1 << cmd2;

  process1->setStandardOutputProcess(process2);

  process1->start("curl",{cmd1});
  process2->start(cmd2);

  process1->waitForStarted();

  process1->waitForFinished();
  process2->waitForFinished();
  Size =process2->readAllStandardOutput();

  qDebug() << Size;
  process2->waitForFinished();

  delete process1;
  delete process2;
}

jsulm

@darongyi Why not simply

process.start("sh", QStringList() << "-c" << "curl \"http://dbpedia.org/data/Haeinsa.json\" | perl dbpediaPlaceData.txt 'Haeinsa'");

?

darongyi

@jsulm said in Can't get value from command chaining ...:

QStringList() << "-c" << "curl "http://dbpedia.org/data/Haeinsa.json" | perl dbpediaPlaceData.txt 'Haeinsa'"

OK.. I think it's simple problem. I will close question..
Thanks to giving answer.