Leaky QProcess output channel - buffer issue

[halfgaar]

Hi,

I'm writing an app that starts a process using QProcess and gives me XML output, which I parse with QXMLStreamreader. Despite using unbuffered output in the external process, there seem to be buffering issues, and we sometimes don't get data.

I found this post, entitled "Qt: Workaround for the leaky QProcess output channel":http://msturm.p7.de/programming_qprocess.php. Is this a known bug? And is there a better workaround than writing to a file?

[SGaist]

Hi,

The best way to verify if it's a known bug/limitation is to go through the "bug report system":http://bugreports.qt-project.org

By the way, which version of Qt are you using ?

[MuldeR]

This is most likely not a bug in QProcess but in the child application you are calling! Explanation: Under Windows, you will notice that, if you do something like printf() in a console application, the output becomes visible in the console immediately. It seems the MSVC runtime automatically disables any "internal" buffering, when connected to a "real" console. But as soon as you redirect the STDOUT of the child application via pipe, this is no longer the case! Now you need to call fflush(stdout), in the child application, in order to make your outputs available. Either that, or you wait until the buffer is full and flushes automatically. But the latter is unreliable and only reveals outputs with a delay! Again: As the buffering is happening inside the child process you cannot do anything from "outside". You need to fix your child application...

Having said that, I use QProcess quite extensively and I have never found it to "lose" any data, as long as the child application is playing nicely.

See also:
http://stackoverflow.com/questions/9037177/windows-console-program-stdout-is-buffered-when-using-pipe-redirection

--

Just for the notes, this is from official x264 code:
@static int64_t print_status( ... )
{
/* [...] */

sprintf( buf, "x264 %d frames: %.2f fps, %.2f kb/s", i_frame, fps, bitrate );
fprintf( stderr, "%s  \r", buf+5 );
x264_cli_set_console_title( buf );
fflush( stderr ); // needed in windows

/* [...] */

}@

http://git.videolan.org/?p=x264.git;a=blob;f=x264.c#l1787

--

About this code:
http://msturm.p7.de/programming_qprocess.php

One problem I see with his "initial solution" is that he is not processing the pending QProcess output after the process has terminated. Connecting to the finished signal and calling printOutput() again might have fixed it.

[halfgaar]

What you're describing is of course a known fenomenon. Our console app uses unbuffered output, and also flushes its buffer. We didn't always do this (we forgot), so we did have more problems before. We thought we fixed it with the unbuffered output, but some weirdness still seems to be happening.

About his initial approach, are you saying that the last bit of data does not trigger the readyRead signal? I mean, he doesn't kill or delete anything, so the QProcess still exists, and the readyRead signals should always fire, right?

If it wouldn't fire after the process is done, the buffer would always be full, because a QProcess doesn't have a flush method.

We're using Qt 5.1, BTW.

[MuldeR]

[quote author="halfgaar" date="1399906336"]What you're describing is of course a known fenomenon. Our console app uses unbuffered output, and also flushes its buffer. We didn't always do this (we forgot), so we did have more problems before. We thought we fixed it with the unbuffered output, but some weirdness still seems to be happening.[/quote]

Okay, so we can exclude that problem.

[quote author="halfgaar" date="1399906336"]About his initial approach, are you saying that the last bit of data does not trigger the readyRead signal? I mean, he doesn't kill or delete anything, so the QProcess still exists, and the readyRead signals should always fire, right?[/quote]

Well, I'm not 100% sure whether readyReadStandardOutput still fires when the process has already terminated. Reading all pending data from the QProcess object after the process has terminated, just to be sure, certainly wouldn't hurt, I suppose. At least I would give it a try in your situation...

I usually do something like:
@while(process.state() != QProcess::NotRunning)
{
if(process.waitForReadyRead(DEADLOCK_TIMEOUT))
{
while(process.canReadLine())
{
output << QString::fromUtf8(process.readLine()).simplified();
}
continue;
}
if(process.state() != QProcess::NotRunning)
{
qWarning("Process encountered a deadlock -> aborting now!");
break;
}
}

process.waitForFinished(FINISH_TIMEOUT);
if(process.state() != QProcess::NotRunning)
{
qWarning("Process still running, going to kill it!");
process.kill();
process.waitForFinished(-1);
}

//Read all pending lines
while(process.canReadLine())
{
output << QString::fromUtf8(process.readLine()).simplified();
}@

Note: Here I did not use Signals&Slots, because the above code was running in a "background" thread anyway. But the idea should be clear.

[halfgaar]

I already do something similar (edit: hmm, it's not really similar. My process.waitforfinished is after the while loop that reads data). It's also not using signals, because it's already in a thread:

@QXmlStreamReader xml;
xml.setDevice(currentProcess.data());

while (!xml.atEnd() && !xml.hasError())
{
QXmlStreamReader::TokenType tokenType = xml.readNext();
bool xmlReadError = false;

while (xml.error() == QXmlStreamReader::PrematureEndOfDocumentError && !xmlReadError)
{
    xmlReadError = ! xml.device()->waitForReadyRead(XML_READNEXT_TIMEOUT);
    tokenType = xml.readNext();
}@

The process is opened as QIODevice::ReadOnly. Tomorrow (I can't right now) I'm going to try QIODevice::ReadOnly | QIODevice::Text. Perhaps that makes a difference.

In any case, I would find it odd that it wouldn't fire a readyRead after the process has terminated, because that would result in a stale buffer.

[MuldeR]

So you are reading the data from your QProcess through a QXmlStreamReader. In this case I would suggest that you try reading the data from the QProcess with a simple QProcess::readLine() loop, like in my example code above, in order to ensure that is not some issue related to the QXmlStreamReader.

BTW: Using QIODevice::Text should make no difference, except that if a "\r\n" sequence is encountered, it will be translated to a single "\n" character.

[halfgaar]

We were also going to try "QIODevice::ReadOnly | QIODevice::Unbuffered", but I suspect that only applies to write devices.

[halfgaar]

OK, the bug was between keyboard and chair. The while loop needed an extra condtion:

@
while ((!xml.atEnd() && !xml.hasError()) || xml.error() == QXmlStreamReader::PrematureEndOfDocumentError)
{
@

It would exit the while loop, and then wait until the process was finished. But, with prematureEndOfDocument, you still want to continue reading.

Glad the hack in the original post wasn't necessary.