Problem processing output from QProcess

RichardC

@jsulm I had tried that, but I gave it another go. Unfortunately it doesn't work. The text doesn't appear to be in the QByteArray to begin with, so the conversion doesn't help.

Out of interest, I wrote a small program on the Mac to output Unicode text:

#include <QCoreApplication>
#include <iostream>

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    QString test = QString::fromUtf8("\t\"description\": \"Импортирован шрифт из\"");
    std::cout << qPrintable(test) << std::endl;

    return a.exec();
}

I then ran that from a QProccess and printed the output with:

qTextEdit->insertPlainText(qProcess.readAllStandardOutput());

In that case read and printed the Russian text without issues, so the above line of code should work and it doesn't appear to be an issue with my program.

Another interesting observation is that the output from versions of MKVToolNix up to v13 will all print correctly when run from a QProccess, but running v14-20 the output cannot be read from the QProccess. That seems to indicate a problem with MKVToolNix, but MKVToolNix v20 will print the output correctly when run in a terminal on the Mac, and if it prints to a terminal you would expect Qt to be able to read the output.

I'm therefore not sure if this is an issue with MKVToolNix or an issue with Qt.

jsulm

@RichardC said in Problem processing output from QProcess:

The text doesn't appear to be in the QByteArray to begin with

You could try to print the length of the array to check that.
But most probably it isn't UTF8 but something else like UTF16 (Windows uses it as far as I know).
I'm quite sure it is encoding problem as it cuts at the first non ASCII character. Unicode characters can contain 0 bytes which are interpreted as "end of string" if you do treat the array as ASCII string. And it looks like exactly that is happening in your case.

JonB

@RichardC
As @jsulm says, 99% certainly a UTF decoding issue from the QByteArray returned from readAllStandardOutput(). You must look at QByteArray::length/size().

I don't know whether the following is unnecessarily complex for your case/MacOS/Russian, but here's what I found I have to use for my readyReadStandardOutput slot (PyQt I'm afraid, but I'm sure you can manage), food for thought:

    def processReadyReadStandardOutput(self):
        # read all output available at this point
        byteArray = self.process.readAllStandardOutput()
        # convert QByteArray to str
        # Linux: the decoder should always be "utf-8"
        # Windows: after *enormous* investigations "utf-8" *mostly* works
        #          but if the output contains a "funny" character like "£"
        #          it will cause a conversion error
        #          then the correct decoder to use is what the command "chcp" says
        #          (e.g. "cp850" for Code Page 850 in UK) to avoid the error *and* correctly display the £
        text = ""
        try:
            try:
                text = byteArray.data().decode('utf-8')
            except UnicodeDecodeError:
                from common import osfunctions
                if osfunctions.isWindows():
                    text = byteArray.data().decode('cp850')
        except:
            # if all this fails for whatever reason (just in case)
            # just output a load of "?"s the length of the output
            # anything is better than throwing an exception here
            text = "?" * len(byteArray)
        self.appendInformativeText(text)

(BTW, if you haven't done so already you'll want to include a call to QProcess::setProcessChannelMode(QProcess::MergedChannels).)

RichardC

@jsulm said in Problem processing output from QProcess:

You could try to print the length of the array to check that.
But most probably it isn't UTF8 but something else like UTF16 (Windows uses it as far as I know).
I'm quite sure it is encoding problem as it cuts at the first non ASCII character. Unicode characters can contain 0 bytes which are interpreted as "end of string" if you do treat the array as ASCII string. And it looks like exactly that is happening in your case.

Both the size() and length() of the QByteArray is 102. The size() and length() of the full output should be 5066.

It therefore seems not to be an encoding issue, just that the data isn't there.

I did try converting it to UTF16 out of interest, but that changed the English text into a series of Chinese characters. It therefore seems that the text is UTF8, and the issue is not the encoding but that the data isn't there.

JonB

@RichardC
That is presumably because signal readyReadStandardOutput and function readAllStandardOutput only get delivered/read whatever happens to be presently available in some buffer, i.e. chunks, not the complete output. It is expected that you will receive the signal multiple times, with subsequent chunks, and you have to handle that.

My processReadyReadStandardOutput() only has to handle a chunk at a time. You may need to append/collect/buffer for your own purposes, depending on what you want to do with the complete output as a whole.

RichardC

@JonB Thanks for the suggestion, but unfortunately that's not the cause of the problem. I am handling the fact that the output is sent in chucks. Some of the output runs to hundreds of thousands of lines, which gets delivered in hundreds of separate chunks. That's not a problem, and the program handles that without issues.

The problem is, once it reaches a unicode character the output ends. No further readyReadStandardOutput signals are emitted so no further output arrives.

I'll edit the first post to try and explain the situation better, because I don't think I did a very good job and I've also found out a few things since then.

jsulm

@RichardC Maybe it's a bug in Qt? You can check Qt bug tracker and file a bug report if there isn't any.

aha_1980

@RichardC said in Problem processing output from QProcess:

Another interesting observation is that the output from versions of MKVToolNix up to v13 will all print correctly when run from a QProccess, but running v14-20 the output cannot be read from the QProccess.

Hmm, is MKVToolNix an open source program? Can you check what was changed from v13 to v14 and could cause that problem?

but MKVToolNix v20 will print the output correctly when run in a terminal on the Mac, and if it prints to a terminal you would expect Qt to be able to read the output.

I would not bet on this. There are programs out there that check if they run in a shell or with redirected output and behave differently then.

As you know which output you expect, and already created a test program you run under QProcess, could you check what happens if you output the expected test from your test program? That way you could verify if it's a Qt problem and already have a test setup for the bugreport.

JonB

@RichardC
As @aha_1980 says, you cannot be sure how a program will behave when launched from a UI vs the command-line/terminal. A couple of thoughts:

I would try running the MKVToolNix program with output redirected to a file and input (probably) closed, as either of these might affect its output.
I would see if the environment variables passed from command-line terminal vs Qt UI are the same or different. Could e.g. LC_LANG be having some effect?

RichardC

@aha_1980 said in Problem processing output from QProcess:

@RichardC said in Problem processing output from QProcess:
Hmm, is MKVToolNix an open source program? Can you check what was changed from v13 to v14 and could cause that problem?

It is open source (source here), but I must confess to finding it difficult to understand. I have asked him, but he doesn't support the MacOS build. I'm going to try it on Linux and see if it has the same problem, since the Linux version is supported.

I would not bet on this. There are programs out there that check if they run in a shell or with redirected output and behave differently then.

The program is intended to be run from a GUI, so I don't think it would do anything unexpected when run from a GUI. On Windows it behaves the same whether it's run from the command line or from a QProccess.

As you know which output you expect, and already created a test program you run under QProcess, could you check what happens if you output the expected test from your test program? That way you could verify if it's a Qt problem and already have a test setup for the bugreport.

I haven't tried outputting the full expected output from the test program, but Qt appears to have no problem reading Unicode characters from the test program. I could try it, but I don't think the same problem would arise. It seems to be a strange incompatibility between Qt and MKVToolNix on the Mac. I'll try it on Linux first, and then I might try doing the full output from a test program.

@JonB said in Problem processing output from QProcess:

@RichardC
As @aha_1980 says, you cannot be sure how a program will behave when launched from a UI vs the command-line/terminal. A couple of thoughts:

I would try running the MKVToolNix program with output redirected to a file and input (probably) closed, as either of these might affect its output.

I tried redirecting the MKVToolNix output to a file, and the full output appears including all Unicode characters.

I would see if the environment variables passed from command-line terminal vs Qt UI are the same or different. Could e.g. LC_LANG be having some effect?

That could possibly be a problem, and I'll have to look into that. For now I'm going to try it on Linux just to see if it works there.

aha_1980

@RichardC

@JonB has put some more interesting question in the ring - especially with the environment variables.

I know for example that LC_NUMERIC=C is set if you run a program in QtCreators debugger (as the stupid GDB otherwise expects locale dependent decimal points).

That of course has an effect on your program and also on all programs started by QProcess.

JonB

@RichardC

I tried redirecting the MKVToolNix output to a file, and the full output appears including all Unicode characters.

You now have a couple of things you can play with, to discover where your actual problem lies:

cat the file in a terminal. Do all the characters display correctly?
Change your QProcess command to cat that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic.
Compare the output bytes in the file against what you see in the readyReadStandardOutput (as far as it goes before getting cut off). Are they identical or is there a difference?

RichardC

@JonB said in Problem processing output from QProcess:

You now have a couple of things you can play with, to discover where your actual problem lies:

cat the file in a terminal. Do all the characters display correctly?

Yes, the full output is displayed along with the Unicode characters.

Change your QProcess command to cat that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic.

Yes, the full contents of the file is returned by readAllStandardOutput(), including the Unicode characters.

Compare the output bytes in the file against what you see in the readyReadStandardOutput (as far as it goes before getting cut off). Are they identical or is there a difference?
I tried the below, where m_file is the contents of the file in a QByteArray. It came out as identical.

void IUIInfoDisplay::OutputText()
{
    QByteArray output = m_qprocMKVToolNix.readAllStandardOutput();

    bool identical = true;
    for (int x = 0 ; x < output.size() ; ++x)
    {
        if (output.at(x) != m_file.at(x))
        {
            QMessageBox::information(this, "Different", "Different", QMessageBox::Ok);
            identical = false;
        }
    }

    if (identical)
        QMessageBox::information(this, "Identical", "Identical", QMessageBox::Ok);
}

I also tried it on Linux (Ubuntu) and like Windows there are no problems reading the output from MKVToolNix.

I think I'll try asking the author of MKVToolNix whether he thinks it's a Qt issue or an MKVToolNix issue. I can easily work around it by sending the output to a temporary file and reading it from there, but it would be nice to know what the cause is.

VRonin

Probably useless suggestion but did you try using QTextStream instead of just accessing the buffer directly?

void IUIInfoDisplay::OutputText()
{
qProcess.setReadChannel(QProcess::StandardOutput);
QTextStream reader(&qProcess);
QString line;
while (reader.readLineInto(&line))
qTextEdit->insertPlainText(line +'\n');
}

JonB

@RichardC

Change your QProcess command to cat that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic.

Yes, the full contents of the file is returned by readAllStandardOutput(), including the Unicode characters.

From what you have said then, since you can read all the exact same characters from full without problem but not from running MKVToolNix, under MacOS only, it appears there is a problem running/reading that from QProcess under MacOS. Which seems a little surprising, but there you are...

Everything points to a "buffering" problem, where you simply do not receive a bunch of further characters from the process after the first block. (When redirected to file, all the characters end up there on file close/termination.) Are you sure you are doing the full ready reads followed by the normal "finished" signal? If I were you, in the "finished" signal I would do an extra "read all output" --- I don't seem to need it under Linux/Windows, but maybe just possibly under MacOS you fail to get the final "ready read" before the "finished". Or, if you do not need to use the signal, there is some other function for "read all output" after the sub-process has finished. This really ought to be the problem...!

RichardC

@VRonin said in Problem processing output from QProcess:

qProcess.setReadChannel(QProcess::StandardOutput);
QTextStream reader(&qProcess);
QString line;
while (reader.readLineInto(&line))
qTextEdit->insertPlainText(line +'\n');

I gave it a try, but the output still terminates when it reaches the first Unicode character.

Thanks anyway for the suggestion.

JonB

@RichardC
If by any chance it is an issue with when the "ready read" signal is delivered versus the "finished" signal, the above will fail in the same way. Do try my last post above, advising an extra read after the "finished" signal...?

RichardC

@JonB said in Problem processing output from QProcess:

From what you have said then, since you can read all the exact same characters from full without problem but not from running MKVToolNix, under MacOS only, it appears there is a problem running/reading that from QProcess under MacOS. Which seems a little surprising, but there you are...

Everything points to a "buffering" problem, where you simply do not receive a bunch of further characters from the process after the first block. (When redirected to file, all the characters end up there on file close/termination.) Are you sure you are doing the full ready reads followed by the normal "finished" signal?

Pretty much certain. If there are no Unicode characters in the ouptut it will read the full output without issues, even when it's hundreds of thousand of lines.

If I were you, in the "finished" signal I would do an extra "read all output" --- I don't seem to need it under Linux/Windows, but maybe just possibly under MacOS you fail to get the final "ready read" before the "finished". Or, if you do not need to use the signal, there is some other function for "read all output" after the sub-process has finished. This really ought to be the problem...!

I tried adding another readAllStandardOutput() in the Finished() function. It returns a QByteArray with a size of 0, so there is no further output.