Problem processing output from QProcess

RichardC

@JonB Thanks for the suggestion, but unfortunately that's not the cause of the problem. I am handling the fact that the output is sent in chucks. Some of the output runs to hundreds of thousands of lines, which gets delivered in hundreds of separate chunks. That's not a problem, and the program handles that without issues.

The problem is, once it reaches a unicode character the output ends. No further readyReadStandardOutput signals are emitted so no further output arrives.

I'll edit the first post to try and explain the situation better, because I don't think I did a very good job and I've also found out a few things since then.

jsulm

@RichardC Maybe it's a bug in Qt? You can check Qt bug tracker and file a bug report if there isn't any.

aha_1980

@RichardC said in Problem processing output from QProcess:

Another interesting observation is that the output from versions of MKVToolNix up to v13 will all print correctly when run from a QProccess, but running v14-20 the output cannot be read from the QProccess.

Hmm, is MKVToolNix an open source program? Can you check what was changed from v13 to v14 and could cause that problem?

but MKVToolNix v20 will print the output correctly when run in a terminal on the Mac, and if it prints to a terminal you would expect Qt to be able to read the output.

I would not bet on this. There are programs out there that check if they run in a shell or with redirected output and behave differently then.

As you know which output you expect, and already created a test program you run under QProcess, could you check what happens if you output the expected test from your test program? That way you could verify if it's a Qt problem and already have a test setup for the bugreport.

JonB

@RichardC
As @aha_1980 says, you cannot be sure how a program will behave when launched from a UI vs the command-line/terminal. A couple of thoughts:

I would try running the MKVToolNix program with output redirected to a file and input (probably) closed, as either of these might affect its output.
I would see if the environment variables passed from command-line terminal vs Qt UI are the same or different. Could e.g. LC_LANG be having some effect?

RichardC

@aha_1980 said in Problem processing output from QProcess:

@RichardC said in Problem processing output from QProcess:
Hmm, is MKVToolNix an open source program? Can you check what was changed from v13 to v14 and could cause that problem?

It is open source (source here), but I must confess to finding it difficult to understand. I have asked him, but he doesn't support the MacOS build. I'm going to try it on Linux and see if it has the same problem, since the Linux version is supported.

I would not bet on this. There are programs out there that check if they run in a shell or with redirected output and behave differently then.

The program is intended to be run from a GUI, so I don't think it would do anything unexpected when run from a GUI. On Windows it behaves the same whether it's run from the command line or from a QProccess.

As you know which output you expect, and already created a test program you run under QProcess, could you check what happens if you output the expected test from your test program? That way you could verify if it's a Qt problem and already have a test setup for the bugreport.

I haven't tried outputting the full expected output from the test program, but Qt appears to have no problem reading Unicode characters from the test program. I could try it, but I don't think the same problem would arise. It seems to be a strange incompatibility between Qt and MKVToolNix on the Mac. I'll try it on Linux first, and then I might try doing the full output from a test program.

@JonB said in Problem processing output from QProcess:

@RichardC
As @aha_1980 says, you cannot be sure how a program will behave when launched from a UI vs the command-line/terminal. A couple of thoughts:

I would try running the MKVToolNix program with output redirected to a file and input (probably) closed, as either of these might affect its output.

I tried redirecting the MKVToolNix output to a file, and the full output appears including all Unicode characters.

I would see if the environment variables passed from command-line terminal vs Qt UI are the same or different. Could e.g. LC_LANG be having some effect?

That could possibly be a problem, and I'll have to look into that. For now I'm going to try it on Linux just to see if it works there.

aha_1980

@RichardC

@JonB has put some more interesting question in the ring - especially with the environment variables.

I know for example that LC_NUMERIC=C is set if you run a program in QtCreators debugger (as the stupid GDB otherwise expects locale dependent decimal points).

That of course has an effect on your program and also on all programs started by QProcess.

JonB

@RichardC

I tried redirecting the MKVToolNix output to a file, and the full output appears including all Unicode characters.

You now have a couple of things you can play with, to discover where your actual problem lies:

cat the file in a terminal. Do all the characters display correctly?
Change your QProcess command to cat that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic.
Compare the output bytes in the file against what you see in the readyReadStandardOutput (as far as it goes before getting cut off). Are they identical or is there a difference?

RichardC

@JonB said in Problem processing output from QProcess:

You now have a couple of things you can play with, to discover where your actual problem lies:

cat the file in a terminal. Do all the characters display correctly?

Yes, the full output is displayed along with the Unicode characters.

Change your QProcess command to cat that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic.

Yes, the full contents of the file is returned by readAllStandardOutput(), including the Unicode characters.

Compare the output bytes in the file against what you see in the readyReadStandardOutput (as far as it goes before getting cut off). Are they identical or is there a difference?
I tried the below, where m_file is the contents of the file in a QByteArray. It came out as identical.

void IUIInfoDisplay::OutputText()
{
    QByteArray output = m_qprocMKVToolNix.readAllStandardOutput();

    bool identical = true;
    for (int x = 0 ; x < output.size() ; ++x)
    {
        if (output.at(x) != m_file.at(x))
        {
            QMessageBox::information(this, "Different", "Different", QMessageBox::Ok);
            identical = false;
        }
    }

    if (identical)
        QMessageBox::information(this, "Identical", "Identical", QMessageBox::Ok);
}

I also tried it on Linux (Ubuntu) and like Windows there are no problems reading the output from MKVToolNix.

I think I'll try asking the author of MKVToolNix whether he thinks it's a Qt issue or an MKVToolNix issue. I can easily work around it by sending the output to a temporary file and reading it from there, but it would be nice to know what the cause is.

VRonin

Probably useless suggestion but did you try using QTextStream instead of just accessing the buffer directly?

void IUIInfoDisplay::OutputText()
{
qProcess.setReadChannel(QProcess::StandardOutput);
QTextStream reader(&qProcess);
QString line;
while (reader.readLineInto(&line))
qTextEdit->insertPlainText(line +'\n');
}

JonB

@RichardC

Change your QProcess command to cat that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic.

Yes, the full contents of the file is returned by readAllStandardOutput(), including the Unicode characters.

From what you have said then, since you can read all the exact same characters from full without problem but not from running MKVToolNix, under MacOS only, it appears there is a problem running/reading that from QProcess under MacOS. Which seems a little surprising, but there you are...

Everything points to a "buffering" problem, where you simply do not receive a bunch of further characters from the process after the first block. (When redirected to file, all the characters end up there on file close/termination.) Are you sure you are doing the full ready reads followed by the normal "finished" signal? If I were you, in the "finished" signal I would do an extra "read all output" --- I don't seem to need it under Linux/Windows, but maybe just possibly under MacOS you fail to get the final "ready read" before the "finished". Or, if you do not need to use the signal, there is some other function for "read all output" after the sub-process has finished. This really ought to be the problem...!

RichardC

@VRonin said in Problem processing output from QProcess:

qProcess.setReadChannel(QProcess::StandardOutput);
QTextStream reader(&qProcess);
QString line;
while (reader.readLineInto(&line))
qTextEdit->insertPlainText(line +'\n');

I gave it a try, but the output still terminates when it reaches the first Unicode character.

Thanks anyway for the suggestion.

JonB

@RichardC
If by any chance it is an issue with when the "ready read" signal is delivered versus the "finished" signal, the above will fail in the same way. Do try my last post above, advising an extra read after the "finished" signal...?

RichardC

@JonB said in Problem processing output from QProcess:

From what you have said then, since you can read all the exact same characters from full without problem but not from running MKVToolNix, under MacOS only, it appears there is a problem running/reading that from QProcess under MacOS. Which seems a little surprising, but there you are...

Everything points to a "buffering" problem, where you simply do not receive a bunch of further characters from the process after the first block. (When redirected to file, all the characters end up there on file close/termination.) Are you sure you are doing the full ready reads followed by the normal "finished" signal?

Pretty much certain. If there are no Unicode characters in the ouptut it will read the full output without issues, even when it's hundreds of thousand of lines.

If I were you, in the "finished" signal I would do an extra "read all output" --- I don't seem to need it under Linux/Windows, but maybe just possibly under MacOS you fail to get the final "ready read" before the "finished". Or, if you do not need to use the signal, there is some other function for "read all output" after the sub-process has finished. This really ought to be the problem...!

I tried adding another readAllStandardOutput() in the Finished() function. It returns a QByteArray with a size of 0, so there is no further output.