QProcess -- working QIODevice::Unbuffered, or adjustable buffersize for faster access to stdout



  • I've written some Qt code that monitors a long-term computation. The calculation is run via a QProcess instance, and outputs a status update each calculation cycle. The update is 1-4 80 char lines, and these happen every 30-60 seconds.

    I need to be able to monitor the status outputs and handle their content quickly (e.g. restart with modified inputs if thing start going badly), but currently QProcess seems to wait until a buffer somewhere fills before making the output available, as it is sometimes several minutes before I see any output, at which point several dozen updates become available at once. When the calculation is run in a terminal, the updates appear smoothly, one at a time.

    Is there some way to get access to the process's stdout sooner? From the docs and a quick skim of the sources, this doesn't look possible at the moment. Does anyone know of a possible way around this? Or can someone confirm that there is no current way to do this so I can open a bug report? ;-)



  • Do you use signals - @readyReadStandardError ()@, @readyReadStandardOutput ()@ ?



  • I'm running QProcess in a background thread, so it's easier to just spin on waitForReadyRead since it won't interfere with the GUI:

    @
    m_process = new QProcess(this);
    m_process->setReadChannel(QProcess::StandardOutput);
    m_process->start(command, QProcess::Unbuffered | QProcess::ReadWrite);
    m_process->waitForStarted(-1);

    while (canGetInput()) {
      m_process->write(getInput());
    }
    
    m_process->closeWriteChannel();
    
    m_process->waitForReadyRead(-1);
    
    while (!m_process->atEnd() || m_process->state() == QProcess::Running) {
      if (m_process->waitForReadyRead(5000)) {
        while (m_process->canReadLine()) {
          parseOutput(QString(m_process->readLine()));
        }
      }
      qDebug() << QString("Current process state: (%1) PID: (%2) error: (%3) bytes Available (%4)")
        .arg(m_process->state())
        .arg(m_process->pid())
        .arg(m_process->error())
        .arg(m_process->bytesAvailable());
    }
    

    @

    bytesAvailable usually returns 0 right until it dumps several minutes worth of data at once.



  • You may use signals and slots in threads.



  • Yes, but that won't solve my problem. The way my code is written it's easier to just spin, rather than do signals/slots. Just out of curiosity, I hooked up a signal to the readRead signal that writes to qDebug when the signal is emitted. The signals aren't emitted any more frequently. I need some way to access the output of the QProcess sooner.



  • Hi dlonie,

    I'm running into exactly the same problem, were you able to figure out anyway to reduce the size of the qprocess buffer, or otherwise get the available output sooner ?



  • Have you read the documentation? It states:

    bq. bool QIODevice::canReadLine () const [virtual]
    Returns true if a complete line of data can be read from the device; otherwise returns false.
    Note that unbuffered devices, which have no way of determining what can be read, always return false.
    This function is often called in conjunction with the readyRead() signal.

    I f you look at the code:
    @
    m_process->start(command, QProcess::Unbuffered | QProcess::ReadWrite);
    @

    Your device is uinbuffered, so canReadLine always returns false....



  • adtrom: Nope -- I think the best solution here would be to add a flush() method to QProcess so that we could pull results as we needed them.

    The trolls claim that this is a user issue, not a bug. It is definitely not a user issue -- there is nothing I can do to get the data faster, I'm stuck waiting on the buffer to fill. I had opened a bug here: http://bugreports.qt.nokia.com/browse/QTBUG-14503 Feel free to add your own comments or open a new one referencing it.

    As for my problem, I just ended up reimplementing the functionality in the external program I was calling. Still working on it now, actually...Sure would have been nice to just be able to call a QProcess!

    Gerolf: Actually it will return true at times (IIRC, Unbuffered mode isn't actually implemented in QProcess, so that flag really doesn't do anything). It was just one of many things I was trying to get this to work. Besides, bytesAvailable() should return non zero in Unbuffered mode when data is available, so I could spin on that instead. In any case, I tried passing every relevant combination of flags I could think of, and none worked the way I needed. There just needs to be a way to flush the pipe.


  • Moderators

    dlonie: Are you sure this is not a user bug? We are using QProcess (via a wrapper) all over the place in Qt Creator and do not have the issue you describe.



  • I'm also using QProcess and catching the StdOut and it works. We also communicate via StdOut / StdIn and it works. That's why I'm abit suprised...



  • Just to be perfectly clear, QProcess does "work" in that it captures everything written on stdout by the external process, the problem is one of timing. In my case the external process writes a single line every few seconds, and I want to get this output as soon as it's available. What happens is that the output is buffered inside QBuffer and my program is not notified that output is available until many lines have been written (note that calling canReadLine explicitly makes no difference). I'm pretty sure that the issue is one of buffering. (caveat: i'm no expert) IE QProcess can set the buffering on the file descriptor it holds to whatever it wants (the fd connected to the external process's stdout). If it's small enough the external process may block more (waiting for QProcess to empty the buffer), but that's what we want in cases where getting the feedback is more important. I would expect the Unbuffered flag to accomplish this, but it seems to have no effect.

    In my case the external program is not one I've written or can recompile, but I've managed to work around the problem by causing it to generate much more output. Now the lines I'm interested in are mixed in with a ton of other crap, but the larger volume of output causes readyRead() to be called more frequently (as the internal QProcess buffer fills up, I'd imagine).

    Hope that clarifies ...



  • Gerolf: That's a good point about Unbuffered and canReadLine(). In fact if Unbuffered were working, you might expect that behavior. IE QProcess would hold perhaps one byte, the external process would block because QProcess's buffer was full, and canReadLine() would never return true (unless that one char happened to be a newline, I guess). So the fact that adding the Unbuffered flag to the start call makes no difference is more evidence that it's broken, in my opinion.

    I think dlonie's suggestion is a good one. If the buffer in QBuffer is large (as now) then the external process won't block, but if we could get any output whenever we wanted by calling flush, we could set the frequency to whatever is appropriate for the latency requirements of our application.



  • Are you really sure that your external process dose write to stdout unbuffered? In many circumstances stdout is bufferd. As long as it is not flushed there is no data for QProcess to read.



  • adtrom has described the problem perfectly, so I won't add much :-)

    I'm just adding that in my case, the external process does provide output immediately when called in a terminal, so there doesn't seem to be any buffering going on outside of Qt.



  • This may seem like a silly question, but have you properly started the thread? I've seen some very strange behavior mixing QProcess with threads, you might be hung up somewhere else.

    Just a thought.



  • dlonie, also redirecting stdout could be of some use



  • @template<class Elem = char, class Tr = std::char_traits<Elem>>

    class StdRedirector : public std::basic_streambuf<Elem, Tr>
    {
    /**

    • Callback Function.
      /
      typedef void (pfncb)(const Elem, std::streamsize _Count, void
      pUsrData);

    public:
    /**

    • Constructor.
    • @param a_Stream the stream to redirect
    • @param a_Cb the callback function
    • @param a_pUsrData user data passed to callback
      /
      StdRedirector(std::ostream& a_Stream, pfncb a_Cb, void
      a_pUsrData):
      m_Stream(a_Stream), m_pCallbackFunction(a_Cb), m_pUserData(a_pUsrData)
      {
      //redirect stream
      m_pBuffer = m_Stream.rdbuf(this);
      };

    /**

    • Destructor.
    • Restores the original stream.
      */
      ~StdRedirector()
      {
      m_Stream.rdbuf(m_pBuffer);
      }

    /**

    • Override xsputn and make it forward data to the callback function.
      /
      std::streamsize xsputn(const Elem
      _Ptr, std::streamsize _Count)
      {
      m_pCallbackFunction(_Ptr, _Count, m_pUserData);
      return _Count;
      }

    /**

    • Override overflow and make it forward data to the callback function.
      */
      typename Tr::int_type overflow(typename Tr::int_type v)
      {
      Elem ch = Tr::to_char_type(v);
      m_pCallbackFunction(&ch, 1, m_pUserData);
      return Tr::not_eof(v);
      }

    protected:
    std::basic_ostream<Elem, Tr>& m_Stream;
    std::streambuf* m_pBuffer;
    pfncb m_pCallbackFunction;
    void* m_pUserData;
    };

    ...

    void outcallback(const char* ptr, std::streamsize count, void* pTextEdit)
    {
    (void) count;
    QTextEdit* p = static_cast<QTextEdit*>(pTextEdit);
    p->append(ptr);
    }
    ...
    QTextEdit *teStdCout = new QTextEdit(wParent);
    m_stdRedirector = new StdRedirector<>(std::cout, outcallback, teStdCout);
    @



  • bump.. i get the same behaviour. the signal readyReadStandardOutput () only appears to be emitted once a Qt buffer somewhere is filled. I know my process is writing frequent lines but get the output in chunks.

    Is there any resolution since 2010?



  • I think the problem is caused by buffering in the external program. Most programs written in C will buffer unless their output stream is a tty. When it isn't, you have to fake it using something like unbuffer (on Unix/Linux - part of the expect package - http://expect.sourceforge.net - which IIUC works by faking tty behaviour).



  • In the QProcess external program, how are you outputting the text to be read in?

    I had a similar issue with my QProcess. When I output to stdout, the readyRead() signal wouldn't emit until after it was already finished since I didn't have to output very much text. I found this link (http://www.lubby.org/ebooks/qtconsoleapp2/qtconsoleapp2.html) to be very helpful for outputting to stdout.

    If you don't want to read the link, stdout (or the QTextStream, if you went that route) has a built-in buffer that needs to be flushed manually to trigger the readyRead() signal. If your output looks like this:

    @QTextStream o(stdout);
    o << "Line of text here";@

    then readyRead() won't trigger unless you do this thousands of times or the QProcess finishes. I've found two options that work, depending on your implementation:

    @QTextStream o(stdout);
    o << "Line of text here" << std::endl;@

    or

    @QTextStream o(stdout);
    o << "Line of text here\n";
    o.flush();@

    Either way, you will trigger the stdout buffer to be flushed and readyRead() should trigger. As I understand it (I'm no expert), "std::endl" is the same as "\n" except that it also triggers a flush in addition to starting a new line. I prefer the second method so that I don't get extra newline characters floating around.

    Hope that helps if anyone runs into this issue.



  • [quote author="Gerolf" date="1291544359"]Have you read the documentation? It states:

    bq. bool QIODevice::canReadLine () const [virtual]
    Returns true if a complete line of data can be read from the device; otherwise returns false.
    Note that unbuffered devices, which have no way of determining what can be read, always return false.
    This function is often called in conjunction with the readyRead() signal.

    I f you look at the code:
    @
    m_process->start(command, QProcess::Unbuffered | QProcess::ReadWrite);
    @

    Your device is uinbuffered, so canReadLine always returns false....[/quote]

    (Though this post quite old) I'd like to address a issue(or share info if possible) on this topic becuase I think I'm solving the similar problem described here.

    QProcess has internal two ring buffers(one for stdout and the other for stderr) of which canReadLine is called and seem to be able to return 'true' even with QProcess's unbuffered flag(At least in my following code). see following QProcess::canReadLine()

    @
    bool QProcess::canReadLine() const
    {
    Q_D(const QProcess);
    const QRingBuffer *readBuffer = (d->processChannel == QProcess::StandardError)
    ? &d->errorReadBuffer
    : &d->outputReadBuffer;
    return readBuffer->canReadLine() || QIODevice::canReadLine();
    }
    @

    I'm having no deep understand of QProcess and QRingBuffer's implementation.
    Here's my code that sends one line of command to spawed process and get one line of response.
    I could get true value from canReadLine and it seems to work as expected. But I'd appreciate if anyone points out any problem in this code nonetheless.

    @
    QString program = QString::fromLocal8Bit("pcomCmd.exe");
    m_process = new QProcess(this);
    m_process->setReadChannel(QProcess::StandardOutput);
    m_process->start(program, QStringList(), QIODevice::ReadWrite | QIODevice::Unbuffered);
    if (m_process->waitForStarted()) {
    connect(m_process, SIGNAL(readyReadStandardOutput()), SLOT(handlePcommCmdOuput()));
    } else {
    killPcommCmd();
    }
    @

    and then call following function to send to command and get response.

    @
    void MainWindow::writeAndReadResponse(const QString& _cmd)
    {
    QString cmd = _cmd + QString("\n");
    m_process->write(cmd.toLocal8Bit());

    m_log->appendPlainText(QString("cmd> ") + cmd);

    const int WAIT_TIME = 1000000;

    QElapsedTimer timer;
    timer.start();
    while (!timer.hasExpired(WAIT_TIME)) {
    m_process->waitForReadyRead(WAIT_TIME-timer.elapsed());
    if (m_process->canReadLine()) {
    QByteArray arr = m_process->readAll();
    m_log->appendPlainText(arr); // HERE I GOT TEXT LINE IN MY QPlainTextEdit widget.

    // do something according to arr

    break;
    }
    }
    }
    @



  • An old thread, but for the record: I had a similar problem with a Python script outputting lines that didn't show up in the QProcess until the python scripot ended. The issue was buffered output in Python. When I start it with the -u flag, it did work.



  • Ran into the same problem. Was running external c code and the readReady() signal would only fire when the StdOut buffer was full enough or after some time eg. 30sek. So i was not recieving the Output in realtime.

    Since i had access to the external code. I disabled buffering for the output.

    [code]
    setbuf(stdout, NULL);
    printf("Testoutput \n");
    [/code]

    after this modification the readReady() Signal fired everytime printf was used in external program.

    regards



  • Same problem met - doing "hcitool -i hci0 lescan" which is quite interactive and does not seem to buffer stdout (e.g. I don't have this problem http://www.spinics.net/lists/linux-bluetooth/msg55523.html while redirecting to file) - I can read data from QProcess once per 4-10 seconds!

    here's code:

    void Tracer::start()
    {
     m_Process = new QProcess();
     m_Process->setReadChannel(QProcess::StandardOutput);
     connect(m_Process, SIGNAL(readyReadStandardOutput()), SLOT(readTracer()));
     QString command = QString("stdbuf -oL hcitool -i hci0 lescan --duplicates");
     m_Process->start(command, QProcess::Unbuffered | QProcess::ReadWrite);
    }
    
    void Tracer::readTracer()
    {
     if (m_Process)
     {
         for (int i=0; i<200; ++i)
         {
             m_Data.append(m_Process->readLine());
             if (m_Data.isEmpty())
            {
               if(i==0)
                 {qDebug()<<"empty"<<QDateTime::currentDateTime().toString(QString("[hh:mm:ss]")).toUtf8(); return;}
                 else
                 {qDebug()<<"last line was "<<i; return;}
             }
             m_Data.clear();
             m_Data.append(m_Process->readLine(18));
                 qDebug()<<m_Data<<QDateTime::currentDateTime().toString(QString("[hh:mm:ss:ms]")).toUtf8();
        }
             m_Data.clear();
     }
    }
    
    

    I started with "hcitool -i hci0 lescan " and then came up to "stdbuf -oL hcitool -i hci0 lescan --duplicates" - no difference



  • looking through "hcitool lescan" source code here http://code.metager.de/source/xref/linux/bluetooth/bluez/tools/hcitool.c#print_advertising_devices

    the output to stdout is made with

    		printf("%s %s\n", addr, name);
    

    that's on line 2471.

    If I write smth like

    std::flush();
    

    or rather fflush(stdout)
    after this line - will this help? or maybe

    setbuf(stdout, NULL); 
    

    ?
    and then I should compile it somehow...



  • fflush(stdout) 
    

    did its job! I receive lines in real-time, lots of times per second.

    Well, of course it'd been oh so much better if QProcess could flush output of its child-process


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.