Solved Problem processing output from QProcess
-
@RichardC
As @jsulm says, 99% certainly a UTF decoding issue from theQByteArray
returned fromreadAllStandardOutput()
. You must look atQByteArray::length/size()
.I don't know whether the following is unnecessarily complex for your case/MacOS/Russian, but here's what I found I have to use for my
readyReadStandardOutput
slot (PyQt I'm afraid, but I'm sure you can manage), food for thought:def processReadyReadStandardOutput(self): # read all output available at this point byteArray = self.process.readAllStandardOutput() # convert QByteArray to str # Linux: the decoder should always be "utf-8" # Windows: after *enormous* investigations "utf-8" *mostly* works # but if the output contains a "funny" character like "£" # it will cause a conversion error # then the correct decoder to use is what the command "chcp" says # (e.g. "cp850" for Code Page 850 in UK) to avoid the error *and* correctly display the £ text = "" try: try: text = byteArray.data().decode('utf-8') except UnicodeDecodeError: from common import osfunctions if osfunctions.isWindows(): text = byteArray.data().decode('cp850') except: # if all this fails for whatever reason (just in case) # just output a load of "?"s the length of the output # anything is better than throwing an exception here text = "?" * len(byteArray) self.appendInformativeText(text)
(BTW, if you haven't done so already you'll want to include a call to
QProcess::setProcessChannelMode(QProcess::MergedChannels)
.) -
@jsulm said in Problem processing output from QProcess:
You could try to print the length of the array to check that.
But most probably it isn't UTF8 but something else like UTF16 (Windows uses it as far as I know).
I'm quite sure it is encoding problem as it cuts at the first non ASCII character. Unicode characters can contain 0 bytes which are interpreted as "end of string" if you do treat the array as ASCII string. And it looks like exactly that is happening in your case.Both the size() and length() of the QByteArray is 102. The size() and length() of the full output should be 5066.
It therefore seems not to be an encoding issue, just that the data isn't there.
I did try converting it to UTF16 out of interest, but that changed the English text into a series of Chinese characters. It therefore seems that the text is UTF8, and the issue is not the encoding but that the data isn't there.
-
@RichardC
That is presumably because signalreadyReadStandardOutput
and functionreadAllStandardOutput
only get delivered/read whatever happens to be presently available in some buffer, i.e. chunks, not the complete output. It is expected that you will receive the signal multiple times, with subsequent chunks, and you have to handle that.My
processReadyReadStandardOutput()
only has to handle a chunk at a time. You may need to append/collect/buffer for your own purposes, depending on what you want to do with the complete output as a whole. -
@JonB Thanks for the suggestion, but unfortunately that's not the cause of the problem. I am handling the fact that the output is sent in chucks. Some of the output runs to hundreds of thousands of lines, which gets delivered in hundreds of separate chunks. That's not a problem, and the program handles that without issues.
The problem is, once it reaches a unicode character the output ends. No further readyReadStandardOutput signals are emitted so no further output arrives.
I'll edit the first post to try and explain the situation better, because I don't think I did a very good job and I've also found out a few things since then.
-
@RichardC Maybe it's a bug in Qt? You can check Qt bug tracker and file a bug report if there isn't any.
-
@RichardC said in Problem processing output from QProcess:
Another interesting observation is that the output from versions of MKVToolNix up to v13 will all print correctly when run from a QProccess, but running v14-20 the output cannot be read from the QProccess.
Hmm, is MKVToolNix an open source program? Can you check what was changed from v13 to v14 and could cause that problem?
but MKVToolNix v20 will print the output correctly when run in a terminal on the Mac, and if it prints to a terminal you would expect Qt to be able to read the output.
I would not bet on this. There are programs out there that check if they run in a shell or with redirected output and behave differently then.
As you know which output you expect, and already created a test program you run under QProcess, could you check what happens if you output the expected test from your test program? That way you could verify if it's a Qt problem and already have a test setup for the bugreport.
-
@RichardC
As @aha_1980 says, you cannot be sure how a program will behave when launched from a UI vs the command-line/terminal. A couple of thoughts:-
I would try running the MKVToolNix program with output redirected to a file and input (probably) closed, as either of these might affect its output.
-
I would see if the environment variables passed from command-line terminal vs Qt UI are the same or different. Could e.g.
LC_LANG
be having some effect?
-
-
@aha_1980 said in Problem processing output from QProcess:
@RichardC said in Problem processing output from QProcess:
Hmm, is MKVToolNix an open source program? Can you check what was changed from v13 to v14 and could cause that problem?It is open source (source here), but I must confess to finding it difficult to understand. I have asked him, but he doesn't support the MacOS build. I'm going to try it on Linux and see if it has the same problem, since the Linux version is supported.
I would not bet on this. There are programs out there that check if they run in a shell or with redirected output and behave differently then.
The program is intended to be run from a GUI, so I don't think it would do anything unexpected when run from a GUI. On Windows it behaves the same whether it's run from the command line or from a QProccess.
As you know which output you expect, and already created a test program you run under QProcess, could you check what happens if you output the expected test from your test program? That way you could verify if it's a Qt problem and already have a test setup for the bugreport.
I haven't tried outputting the full expected output from the test program, but Qt appears to have no problem reading Unicode characters from the test program. I could try it, but I don't think the same problem would arise. It seems to be a strange incompatibility between Qt and MKVToolNix on the Mac. I'll try it on Linux first, and then I might try doing the full output from a test program.
@JonB said in Problem processing output from QProcess:
@RichardC
As @aha_1980 says, you cannot be sure how a program will behave when launched from a UI vs the command-line/terminal. A couple of thoughts:- I would try running the MKVToolNix program with output redirected to a file and input (probably) closed, as either of these might affect its output.
I tried redirecting the MKVToolNix output to a file, and the full output appears including all Unicode characters.
- I would see if the environment variables passed from command-line terminal vs Qt UI are the same or different. Could e.g.
LC_LANG
be having some effect?
That could possibly be a problem, and I'll have to look into that. For now I'm going to try it on Linux just to see if it works there.
-
@JonB has put some more interesting question in the ring - especially with the environment variables.
I know for example that
LC_NUMERIC=C
is set if you run a program in QtCreators debugger (as the stupid GDB otherwise expects locale dependent decimal points).That of course has an effect on your program and also on all programs started by QProcess.
-
I tried redirecting the MKVToolNix output to a file, and the full output appears including all Unicode characters.
You now have a couple of things you can play with, to discover where your actual problem lies:
cat
the file in a terminal. Do all the characters display correctly?- Change your
QProcess
command tocat
that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic. - Compare the output bytes in the file against what you see in the
readyReadStandardOutput
(as far as it goes before getting cut off). Are they identical or is there a difference?
-
@JonB said in Problem processing output from QProcess:
You now have a couple of things you can play with, to discover where your actual problem lies:
cat
the file in a terminal. Do all the characters display correctly?
Yes, the full output is displayed along with the Unicode characters.
- Change your
QProcess
command tocat
that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic.
Yes, the full contents of the file is returned by
readAllStandardOutput()
, including the Unicode characters.- Compare the output bytes in the file against what you see in the
readyReadStandardOutput
(as far as it goes before getting cut off). Are they identical or is there a difference?
I tried the below, where m_file is the contents of the file in a QByteArray. It came out as identical.
void IUIInfoDisplay::OutputText() { QByteArray output = m_qprocMKVToolNix.readAllStandardOutput(); bool identical = true; for (int x = 0 ; x < output.size() ; ++x) { if (output.at(x) != m_file.at(x)) { QMessageBox::information(this, "Different", "Different", QMessageBox::Ok); identical = false; } } if (identical) QMessageBox::information(this, "Identical", "Identical", QMessageBox::Ok); }
I also tried it on Linux (Ubuntu) and like Windows there are no problems reading the output from MKVToolNix.
I think I'll try asking the author of MKVToolNix whether he thinks it's a Qt issue or an MKVToolNix issue. I can easily work around it by sending the output to a temporary file and reading it from there, but it would be nice to know what the cause is.
-
Probably useless suggestion but did you try using
QTextStream
instead of just accessing the buffer directly?void IUIInfoDisplay::OutputText() { qProcess.setReadChannel(QProcess::StandardOutput); QTextStream reader(&qProcess); QString line; while (reader.readLineInto(&line)) qTextEdit->insertPlainText(line +'\n'); }
-
Change your QProcess command to cat that file. Do you get the output bytes back correctly or not? This tells you whether it's running the MKVToolNix sub-process or whether it's the content of the output which is problematic.
Yes, the full contents of the file is returned by readAllStandardOutput(), including the Unicode characters.
From what you have said then, since you can read all the exact same characters from full without problem but not from running MKVToolNix, under MacOS only, it appears there is a problem running/reading that from
QProcess
under MacOS. Which seems a little surprising, but there you are...Everything points to a "buffering" problem, where you simply do not receive a bunch of further characters from the process after the first block. (When redirected to file, all the characters end up there on file close/termination.) Are you sure you are doing the full ready reads followed by the normal "finished" signal? If I were you, in the "finished" signal I would do an extra "read all output" --- I don't seem to need it under Linux/Windows, but maybe just possibly under MacOS you fail to get the final "ready read" before the "finished". Or, if you do not need to use the signal, there is some other function for "read all output" after the sub-process has finished. This really ought to be the problem...!
-
@VRonin said in Problem processing output from QProcess:
qProcess.setReadChannel(QProcess::StandardOutput);
QTextStream reader(&qProcess);
QString line;
while (reader.readLineInto(&line))
qTextEdit->insertPlainText(line +'\n');I gave it a try, but the output still terminates when it reaches the first Unicode character.
Thanks anyway for the suggestion.
-
@RichardC
If by any chance it is an issue with when the "ready read" signal is delivered versus the "finished" signal, the above will fail in the same way. Do try my last post above, advising an extra read after the "finished" signal...? -
@JonB said in Problem processing output from QProcess:
From what you have said then, since you can read all the exact same characters from full without problem but not from running MKVToolNix, under MacOS only, it appears there is a problem running/reading that from
QProcess
under MacOS. Which seems a little surprising, but there you are...Everything points to a "buffering" problem, where you simply do not receive a bunch of further characters from the process after the first block. (When redirected to file, all the characters end up there on file close/termination.) Are you sure you are doing the full ready reads followed by the normal "finished" signal?
Pretty much certain. If there are no Unicode characters in the ouptut it will read the full output without issues, even when it's hundreds of thousand of lines.
If I were you, in the "finished" signal I would do an extra "read all output" --- I don't seem to need it under Linux/Windows, but maybe just possibly under MacOS you fail to get the final "ready read" before the "finished". Or, if you do not need to use the signal, there is some other function for "read all output" after the sub-process has finished. This really ought to be the problem...!
I tried adding another
readAllStandardOutput()
in the Finished() function. It returns a QByteArray with a size of 0, so there is no further output.