Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. QIODevice::bytesWritten but bytes are not written on asynchronous devices
Forum Updated to NodeBB v4.3 + New Features

QIODevice::bytesWritten but bytes are not written on asynchronous devices

Scheduled Pinned Locked Moved Unsolved General and Desktop
13 Posts 4 Posters 4.0k Views 3 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • miclandM Offline
    miclandM Offline
    micland
    wrote on last edited by
    #2

    If I'm unterstanding you right your Qt program generates data that you write to the stdin of a separate process. In my opinion there should be a flow control provided by the operating system so that you can't write faster than the other process can read. So I don't see a reason for throtteling your write method. QIODevice::write(...) should block until all data is written. And afterwards (and perhaps during this call) the bytesWritten signal is emitted to inform you about the progress... But the signal says bytesWritten - not bytesToWrite ;-)

    ThierryLelegardT 1 Reply Last reply
    0
    • miclandM micland

      If I'm unterstanding you right your Qt program generates data that you write to the stdin of a separate process. In my opinion there should be a flow control provided by the operating system so that you can't write faster than the other process can read. So I don't see a reason for throtteling your write method. QIODevice::write(...) should block until all data is written. And afterwards (and perhaps during this call) the bytesWritten signal is emitted to inform you about the progress... But the signal says bytesWritten - not bytesToWrite ;-)

      ThierryLelegardT Offline
      ThierryLelegardT Offline
      ThierryLelegard
      wrote on last edited by
      #3

      @micland There is a flow control at operating system level, indeed. But this is off topic, here the problem is a Qt one.

      Qt precisely wants to avoid to be blocked by the OS flow control, blocked on OS calls which would degrade the responsiveness of the event loop. Briefly looking at Qt source code, when you call QIODevice::write on a QProcess, the data are not immediately written to the device. Instead, Qt accumulates the data into an internal QRingBuffer (a Qt internal class) until the OS device is ready to accept data without blocking. This is why the virtual memory of the process increases.

      In other words, Qt replaces the system flow control by its own.

      The Qt documentation seems to indicate that bytesWritten is emitted when the data are actually written to the OS device, freeing the equivalent amount in the Qt internal QRingBuffer.

      This is a perfectly decent architecture (like most things in Qt), except that it does not seem to work like that.

      -Thierry

      miclandM 1 Reply Last reply
      0
      • ThierryLelegardT ThierryLelegard

        @micland There is a flow control at operating system level, indeed. But this is off topic, here the problem is a Qt one.

        Qt precisely wants to avoid to be blocked by the OS flow control, blocked on OS calls which would degrade the responsiveness of the event loop. Briefly looking at Qt source code, when you call QIODevice::write on a QProcess, the data are not immediately written to the device. Instead, Qt accumulates the data into an internal QRingBuffer (a Qt internal class) until the OS device is ready to accept data without blocking. This is why the virtual memory of the process increases.

        In other words, Qt replaces the system flow control by its own.

        The Qt documentation seems to indicate that bytesWritten is emitted when the data are actually written to the OS device, freeing the equivalent amount in the Qt internal QRingBuffer.

        This is a perfectly decent architecture (like most things in Qt), except that it does not seem to work like that.

        -Thierry

        miclandM Offline
        miclandM Offline
        micland
        wrote on last edited by
        #4

        @ThierryLelegard I did not take a look at the Qt code but I would expect that the mentioned QRingBuffer is just a buffer with a specified (max) size and will not increase to unlimited. Are you sure that the observerd memory consumption is caused by this buffer and not anywhere else in your code?

        ThierryLelegardT 1 Reply Last reply
        0
        • miclandM micland

          @ThierryLelegard I did not take a look at the Qt code but I would expect that the mentioned QRingBuffer is just a buffer with a specified (max) size and will not increase to unlimited. Are you sure that the observerd memory consumption is caused by this buffer and not anywhere else in your code?

          ThierryLelegardT Offline
          ThierryLelegardT Offline
          ThierryLelegard
          wrote on last edited by
          #5

          @micland The problem is definitely in QProcess. I forgot to mention that if I prematurely stop the data generation, the virtual memory is immediately freed when I delete the QProcess.

          1 Reply Last reply
          0
          • K Offline
            K Offline
            kuzulis
            Qt Champions 2020
            wrote on last edited by
            #6

            bytesWritten() should emitted after data were really send. but bytesWritten() is implemented with different ways for different I/O classes... so, it has different behavior.. unfortunately.

            I mean somewhere it is emitted after the write() directly (that is not a right way, IMHO), but somewhere it emitted after the kernel notified that TX FIFO is empty (that is a right way, IMHO).

            ThierryLelegardT 1 Reply Last reply
            0
            • K kuzulis

              bytesWritten() should emitted after data were really send. but bytesWritten() is implemented with different ways for different I/O classes... so, it has different behavior.. unfortunately.

              I mean somewhere it is emitted after the write() directly (that is not a right way, IMHO), but somewhere it emitted after the kernel notified that TX FIFO is empty (that is a right way, IMHO).

              ThierryLelegardT Offline
              ThierryLelegardT Offline
              ThierryLelegard
              wrote on last edited by
              #7

              @kuzulis I do agree with you about what should be the right way.

              Even if bytesWritten is not emitted when the kernel TX fifo is empty, that could be ok if it is at least emitted when data leave the user space (ie. after the system call write(2) on UNIX, not the QIODevice::write method). Because what we need to know is when there is some free space in virtual memory to generate more data.

              With QProcess, and maybe others, bytesWritten is clearly emitted when the data are still in user space, consuming virtual memory. This means that bytesWritten cannot be used as a reliable flow control mechanism without filling the process virtual memory.

              Now the problem is how should we do? I posted this to collect feedback and experience from others. Having a fast producer and a slow consumer is a quite common pattern. Some other people should have run into this in Qt applications, at least with QTcpSocket if not QProcess. How did you achieve this? Anyone? No one really?

              -Thierry

              kshegunovK 1 Reply Last reply
              0
              • ThierryLelegardT ThierryLelegard

                @kuzulis I do agree with you about what should be the right way.

                Even if bytesWritten is not emitted when the kernel TX fifo is empty, that could be ok if it is at least emitted when data leave the user space (ie. after the system call write(2) on UNIX, not the QIODevice::write method). Because what we need to know is when there is some free space in virtual memory to generate more data.

                With QProcess, and maybe others, bytesWritten is clearly emitted when the data are still in user space, consuming virtual memory. This means that bytesWritten cannot be used as a reliable flow control mechanism without filling the process virtual memory.

                Now the problem is how should we do? I posted this to collect feedback and experience from others. Having a fast producer and a slow consumer is a quite common pattern. Some other people should have run into this in Qt applications, at least with QTcpSocket if not QProcess. How did you achieve this? Anyone? No one really?

                -Thierry

                kshegunovK Offline
                kshegunovK Offline
                kshegunov
                Moderators
                wrote on last edited by kshegunov
                #8

                @ThierryLelegard
                From what I can see the signal is raised after windows notifies Qt about the bytes departing onto their merry way. I haven't closely inspected the code, I only glanced at it, but this is how it looks to me.

                Even if bytesWritten is not emitted when the kernel TX fifo is empty

                The standard input/output isn't much different from a file and to be honest I see no reason the signal to be emitted after all the input is consumed. I'd always expect the signal to be emitted after the writing has been done, not after someone consumed what was written on the other end.

                Some other people should have run into this in Qt applications, at least with QTcpSocket if not QProcess.

                I don't see the connection. The socket lives in your address space and you can have full control over it, not exactly true for a process.

                I posted this to collect feedback and experience from others.

                Is the process you're invoking the standard ffmpeg, or do you have some wrapper around it? One thing that you'd ordinarily try is to signal the data availability through a system semaphore. However, not knowing your setup in details, this is a somewhat far-fetched advice.

                Kind regards.

                Read and abide by the Qt Code of Conduct

                1 Reply Last reply
                0
                • K Offline
                  K Offline
                  kuzulis
                  Qt Champions 2020
                  wrote on last edited by kuzulis
                  #9

                  I'd always expect the signal to be emitted after the writing has been done, not after someone consumed what was written on the other end.

                  I'm personally disagree with this.

                  1. See, if we do emit bytesWritten() after we call system write(), then the method waitForBytesWritten() has not sense! Because bytesWritten() will be emitted immediatelly.

                  2. For some "slow" devices, e.g. as serial port, we need to expect that bytesWritten() means that data will be written physically (of course, kernel's Tx fifo does not guarantees it, but it is near to ideal)

                  And same with the Windows API: we need to emit bytesWritten() only when async operation has been completed, e.g. after kernel's IOCP notified us, but not after WriteFile..

                  kshegunovK 1 Reply Last reply
                  0
                  • K kuzulis

                    I'd always expect the signal to be emitted after the writing has been done, not after someone consumed what was written on the other end.

                    I'm personally disagree with this.

                    1. See, if we do emit bytesWritten() after we call system write(), then the method waitForBytesWritten() has not sense! Because bytesWritten() will be emitted immediatelly.

                    2. For some "slow" devices, e.g. as serial port, we need to expect that bytesWritten() means that data will be written physically (of course, kernel's Tx fifo does not guarantees it, but it is near to ideal)

                    And same with the Windows API: we need to emit bytesWritten() only when async operation has been completed, e.g. after kernel's IOCP notified us, but not after WriteFile..

                    kshegunovK Offline
                    kshegunovK Offline
                    kshegunov
                    Moderators
                    wrote on last edited by
                    #10

                    I'm personally disagree with this.

                    @kuzulis, I'm by no means last authority and I'm not a Qt dev. I'm just sharing what I would expect from the API, I'm not arguing that this is the way it's supposed to be implemented.

                    See, if we do emit bytesWritten() after we call system write(), then the method waitForBytesWritten() has not sense! Because bytesWritten() will be emitted immediatelly.

                    I can relate to that. However, it may not be possible to know when the OS' buffer is flushed (thus when the bytes are actually written), but In any case OS buffering is (mostly) something beyond the user programmers control. Still, it appears (the reason I linked the relevant part of the Qt source in my previous post) that the Qt devs actually wait for the OS to report the bytes written before emitting the signal (at least for the windows implementation of QProcess).

                    For some "slow" devices, e.g. as serial port, we need to expect that bytesWritten() means that data will be written physically (of course, kernel's Tx fifo does not guarantees it, but it is near to ideal)

                    That's what I mean. You can push things to the OS, but in the best case you depend on it to notify you on the status of that written data.

                    And same with the Windows API: we need to emit bytesWritten() only when async operation has been completed, e.g. after kernel's IOCP notified us, but not after WriteFile..

                    Take a look at the source. I think this is the whole point of using the overlapped structure and emitting the signal when said structure is signaled.

                    Kind regards.

                    Read and abide by the Qt Code of Conduct

                    1 Reply Last reply
                    0
                    • ThierryLelegardT Offline
                      ThierryLelegardT Offline
                      ThierryLelegard
                      wrote on last edited by
                      #11

                      Finally, this is a Qt bug on Windows.

                      On Linux, QProcess::bytesWritten works as expected, when the data have left the application's memory. Whether the signal is emitted after the write(2) system call or when the kernel fifo is empty is not important for the application to be able to regulate the flow.

                      The problem on Windows is that the signal is emitted immediately after QProcess::write, before - and even long before - the data is sent to the kernel.

                      Conclusion: on Windows the fast producer / slow consumer pattern is impossible to implement in a reliable way.

                      I reported this as https://bugreports.qt.io/browse/QTBUG-54118

                      1 Reply Last reply
                      0
                      • ThierryLelegardT Offline
                        ThierryLelegardT Offline
                        ThierryLelegard
                        wrote on last edited by
                        #12

                        For the record, this is fixed in Qt 5.7.0.

                        The same application that blew up from a memory exhaustion with Qt 5.6 on Windows is now perfectly stable and the flow regulation based on QProcess::bytesWritten works as expected.

                        kshegunovK 1 Reply Last reply
                        2
                        • ThierryLelegardT ThierryLelegard

                          For the record, this is fixed in Qt 5.7.0.

                          The same application that blew up from a memory exhaustion with Qt 5.6 on Windows is now perfectly stable and the flow regulation based on QProcess::bytesWritten works as expected.

                          kshegunovK Offline
                          kshegunovK Offline
                          kshegunov
                          Moderators
                          wrote on last edited by
                          #13

                          @ThierryLelegard
                          Thanks for sharing. It'd be really interesting to know what caused it in the first place. Did you make a bisect as suggested in the bug report page?

                          Read and abide by the Qt Code of Conduct

                          1 Reply Last reply
                          0

                          • Login

                          • Login or register to search.
                          • First post
                            Last post
                          0
                          • Categories
                          • Recent
                          • Tags
                          • Popular
                          • Users
                          • Groups
                          • Search
                          • Get Qt Extensions
                          • Unsolved