QIODevice::bytesWritten but bytes are not written on asynchronous devices
-
Hi,
On an asynchronous QIODevice (here a QProcess), I suspect that QIODevice::bytesWritten is emitted before the data are actually sent on the device. Could this be possible?
Here is the situation. On Windows, I create a QProcess with a command which consumes its standard input at slow speed (this is an instance of ffmpeg doing video conversion). The Qt application which created the QProcess extracts video data from somewhere and writes them on the write channel of the QProcess (ie. the standard input of the process). But the Qt application produces video data a lot faster than the ffmpeg process can possibly consume them.
To regulate the traffic and avoid an accumulation of data inside the memory of the Qt application, a slot is connected to QIODevice::bytesWritten on the QProcess. The application accumulates the total size of the internally generated data and all written data (sum of all QIODevice::bytesWritten). The difference between the two should represent the data in memory. When this difference is larger than a given threshold, the application stops generating data. Data generation will restart later, after some QIODevice::bytesWritten, when the "data in memory" go below the threshold.
However, it does not work. QIODevice::bytesWritten is always reported very rapidly, as if all write operations were immediate. The difference between the generated data and QIODevice::bytesWritten is always zero. Using the Windows' Resource Monitor, I can see that the "Private Space" of the Qt application grows very fast and eventually crashes the application due to memory exhaustion.
So, how could I be notified that the written data are really written, gone, away, using no more memory in the Qt process?
Or, could you suggest another regulation method for such a scenario?
The application does many things in parallel from the event loop, so any form of "wait for" is excluded. I need a notification, typically a signal.
I assume that QProcess and QTcpSocket have similar behaviors, both being documented as asynchronous. I also assume that streaming data to a socket faster than the receiver can read is a common problem. How is it usually solved?
Thanks in advance.
-Thierry -
If I'm unterstanding you right your Qt program generates data that you write to the stdin of a separate process. In my opinion there should be a flow control provided by the operating system so that you can't write faster than the other process can read. So I don't see a reason for throtteling your write method.
QIODevice::write(...)should block until all data is written. And afterwards (and perhaps during this call) thebytesWrittensignal is emitted to inform you about the progress... But the signal says bytesWritten - not bytesToWrite ;-) -
If I'm unterstanding you right your Qt program generates data that you write to the stdin of a separate process. In my opinion there should be a flow control provided by the operating system so that you can't write faster than the other process can read. So I don't see a reason for throtteling your write method.
QIODevice::write(...)should block until all data is written. And afterwards (and perhaps during this call) thebytesWrittensignal is emitted to inform you about the progress... But the signal says bytesWritten - not bytesToWrite ;-)@micland There is a flow control at operating system level, indeed. But this is off topic, here the problem is a Qt one.
Qt precisely wants to avoid to be blocked by the OS flow control, blocked on OS calls which would degrade the responsiveness of the event loop. Briefly looking at Qt source code, when you call QIODevice::write on a QProcess, the data are not immediately written to the device. Instead, Qt accumulates the data into an internal QRingBuffer (a Qt internal class) until the OS device is ready to accept data without blocking. This is why the virtual memory of the process increases.
In other words, Qt replaces the system flow control by its own.
The Qt documentation seems to indicate that bytesWritten is emitted when the data are actually written to the OS device, freeing the equivalent amount in the Qt internal QRingBuffer.
This is a perfectly decent architecture (like most things in Qt), except that it does not seem to work like that.
-Thierry
-
@micland There is a flow control at operating system level, indeed. But this is off topic, here the problem is a Qt one.
Qt precisely wants to avoid to be blocked by the OS flow control, blocked on OS calls which would degrade the responsiveness of the event loop. Briefly looking at Qt source code, when you call QIODevice::write on a QProcess, the data are not immediately written to the device. Instead, Qt accumulates the data into an internal QRingBuffer (a Qt internal class) until the OS device is ready to accept data without blocking. This is why the virtual memory of the process increases.
In other words, Qt replaces the system flow control by its own.
The Qt documentation seems to indicate that bytesWritten is emitted when the data are actually written to the OS device, freeing the equivalent amount in the Qt internal QRingBuffer.
This is a perfectly decent architecture (like most things in Qt), except that it does not seem to work like that.
-Thierry
@ThierryLelegard I did not take a look at the Qt code but I would expect that the mentioned QRingBuffer is just a buffer with a specified (max) size and will not increase to unlimited. Are you sure that the observerd memory consumption is caused by this buffer and not anywhere else in your code?
-
@ThierryLelegard I did not take a look at the Qt code but I would expect that the mentioned QRingBuffer is just a buffer with a specified (max) size and will not increase to unlimited. Are you sure that the observerd memory consumption is caused by this buffer and not anywhere else in your code?
@micland The problem is definitely in QProcess. I forgot to mention that if I prematurely stop the data generation, the virtual memory is immediately freed when I delete the QProcess.
-
bytesWritten() should emitted after data were really send. but bytesWritten() is implemented with different ways for different I/O classes... so, it has different behavior.. unfortunately.
I mean somewhere it is emitted after the write() directly (that is not a right way, IMHO), but somewhere it emitted after the kernel notified that TX FIFO is empty (that is a right way, IMHO).
-
bytesWritten() should emitted after data were really send. but bytesWritten() is implemented with different ways for different I/O classes... so, it has different behavior.. unfortunately.
I mean somewhere it is emitted after the write() directly (that is not a right way, IMHO), but somewhere it emitted after the kernel notified that TX FIFO is empty (that is a right way, IMHO).
@kuzulis I do agree with you about what should be the right way.
Even if bytesWritten is not emitted when the kernel TX fifo is empty, that could be ok if it is at least emitted when data leave the user space (ie. after the system call write(2) on UNIX, not the QIODevice::write method). Because what we need to know is when there is some free space in virtual memory to generate more data.
With QProcess, and maybe others, bytesWritten is clearly emitted when the data are still in user space, consuming virtual memory. This means that bytesWritten cannot be used as a reliable flow control mechanism without filling the process virtual memory.
Now the problem is how should we do? I posted this to collect feedback and experience from others. Having a fast producer and a slow consumer is a quite common pattern. Some other people should have run into this in Qt applications, at least with QTcpSocket if not QProcess. How did you achieve this? Anyone? No one really?
-Thierry
-
@kuzulis I do agree with you about what should be the right way.
Even if bytesWritten is not emitted when the kernel TX fifo is empty, that could be ok if it is at least emitted when data leave the user space (ie. after the system call write(2) on UNIX, not the QIODevice::write method). Because what we need to know is when there is some free space in virtual memory to generate more data.
With QProcess, and maybe others, bytesWritten is clearly emitted when the data are still in user space, consuming virtual memory. This means that bytesWritten cannot be used as a reliable flow control mechanism without filling the process virtual memory.
Now the problem is how should we do? I posted this to collect feedback and experience from others. Having a fast producer and a slow consumer is a quite common pattern. Some other people should have run into this in Qt applications, at least with QTcpSocket if not QProcess. How did you achieve this? Anyone? No one really?
-Thierry
@ThierryLelegard
From what I can see the signal is raised after windows notifies Qt about the bytes departing onto their merry way. I haven't closely inspected the code, I only glanced at it, but this is how it looks to me.Even if bytesWritten is not emitted when the kernel TX fifo is empty
The standard input/output isn't much different from a file and to be honest I see no reason the signal to be emitted after all the input is consumed. I'd always expect the signal to be emitted after the writing has been done, not after someone consumed what was written on the other end.
Some other people should have run into this in Qt applications, at least with QTcpSocket if not QProcess.
I don't see the connection. The socket lives in your address space and you can have full control over it, not exactly true for a process.
I posted this to collect feedback and experience from others.
Is the process you're invoking the standard
ffmpeg, or do you have some wrapper around it? One thing that you'd ordinarily try is to signal the data availability through a system semaphore. However, not knowing your setup in details, this is a somewhat far-fetched advice.Kind regards.
-
I'd always expect the signal to be emitted after the writing has been done, not after someone consumed what was written on the other end.
I'm personally disagree with this.
-
See, if we do emit bytesWritten() after we call system write(), then the method waitForBytesWritten() has not sense! Because bytesWritten() will be emitted immediatelly.
-
For some "slow" devices, e.g. as serial port, we need to expect that bytesWritten() means that data will be written physically (of course, kernel's Tx fifo does not guarantees it, but it is near to ideal)
And same with the Windows API: we need to emit bytesWritten() only when async operation has been completed, e.g. after kernel's IOCP notified us, but not after WriteFile..
-
-
I'd always expect the signal to be emitted after the writing has been done, not after someone consumed what was written on the other end.
I'm personally disagree with this.
-
See, if we do emit bytesWritten() after we call system write(), then the method waitForBytesWritten() has not sense! Because bytesWritten() will be emitted immediatelly.
-
For some "slow" devices, e.g. as serial port, we need to expect that bytesWritten() means that data will be written physically (of course, kernel's Tx fifo does not guarantees it, but it is near to ideal)
And same with the Windows API: we need to emit bytesWritten() only when async operation has been completed, e.g. after kernel's IOCP notified us, but not after WriteFile..
I'm personally disagree with this.
@kuzulis, I'm by no means last authority and I'm not a Qt dev. I'm just sharing what I would expect from the API, I'm not arguing that this is the way it's supposed to be implemented.
See, if we do emit bytesWritten() after we call system write(), then the method waitForBytesWritten() has not sense! Because bytesWritten() will be emitted immediatelly.
I can relate to that. However, it may not be possible to know when the OS' buffer is flushed (thus when the bytes are actually written), but In any case OS buffering is (mostly) something beyond the user programmers control. Still, it appears (the reason I linked the relevant part of the Qt source in my previous post) that the Qt devs actually wait for the OS to report the bytes written before emitting the signal (at least for the windows implementation of
QProcess).For some "slow" devices, e.g. as serial port, we need to expect that bytesWritten() means that data will be written physically (of course, kernel's Tx fifo does not guarantees it, but it is near to ideal)
That's what I mean. You can push things to the OS, but in the best case you depend on it to notify you on the status of that written data.
And same with the Windows API: we need to emit bytesWritten() only when async operation has been completed, e.g. after kernel's IOCP notified us, but not after WriteFile..
Take a look at the source. I think this is the whole point of using the overlapped structure and emitting the signal when said structure is signaled.
Kind regards.
-
-
Finally, this is a Qt bug on Windows.
On Linux, QProcess::bytesWritten works as expected, when the data have left the application's memory. Whether the signal is emitted after the write(2) system call or when the kernel fifo is empty is not important for the application to be able to regulate the flow.
The problem on Windows is that the signal is emitted immediately after QProcess::write, before - and even long before - the data is sent to the kernel.
Conclusion: on Windows the fast producer / slow consumer pattern is impossible to implement in a reliable way.
I reported this as https://bugreports.qt.io/browse/QTBUG-54118
-
For the record, this is fixed in Qt 5.7.0.
The same application that blew up from a memory exhaustion with Qt 5.6 on Windows is now perfectly stable and the flow regulation based on QProcess::bytesWritten works as expected.
-
For the record, this is fixed in Qt 5.7.0.
The same application that blew up from a memory exhaustion with Qt 5.6 on Windows is now perfectly stable and the flow regulation based on QProcess::bytesWritten works as expected.
@ThierryLelegard
Thanks for sharing. It'd be really interesting to know what caused it in the first place. Did you make a bisect as suggested in the bug report page?
