How to Display Frames in QML from C++ for Real-Time Applications

[GrecKo]

For a real time image stream I would use VideoOutput + QVideoSink instead of QQuickImageProvider.

[Wertyus]

@GrecKo I'm following your advice, but I'm unable to find the "videoOutput" element I created in QML from C++. Do you have any suggestions for this?

[Ronel_qtmaster]

@Wertyus Hi . Lets make some things clear. For both Your Device and User Interface you have to implement the UDP protocol to send and receive frames.
Now in your device, create a QAbstractVideoFilter to detect frames in realtime and couple it with a VideoOutput on the qml side.
Create a Class inherited from QThread that will take each frames from the filter, and send it to the Server using QUdpSocket.
On the server side (Your GUI app) create a QUdpSocket listenning to the incomming frames.
if you have high resolution images, you might consider scaling or compressing them. Hope it helps

[Wertyus]

@Ronel_qtmaster @GrecKo @jsulm

Hello,

I have resolved the issue. I am now temporarily saving a few frames to a directory on the computer and updating the Image element's source in QML with these frames. Currently, I am working on achieving smoother transitions, but so far, this is the only method that has worked reliably.

Since UDP management is handled by the device's SDK, I couldn't use Qt Creator's UDP library. Additionally, I encountered an issue where the signal-slot connections couldn't find the QML element. When trying the "VideoOutput + QVideoSink" approach, the VideoOutput element was not recognized at all.

I still haven't figured out the reasons behind these issues, but at least I now have a working method. If you have any recommendations regarding these topics, I'd be happy to hear them. Otherwise, I just wanted to thank you for your support and wish you all the best in your work!

Best regards,

[JoeCFD]

@Wertyus Maybe take a look at gstreamer and FFmpeg to handle this. Use raw gstreamer or FFmpeg code to catch the streaming and render it in qml sink in gstreamer. You can test with Gstreamer or FFmpeg command-line code and then convert it into C++ code.

Qt multimedia module is not ideal for this type of app.

[Bob64]

I don't know if it helps, but I have done something that involves rendering a stream of images as an animated view in QML.

I simply use a class derived from QQuickPaintedItem that I expose to QML. This incorporates a timer that checks for the availability of a new image when it times out. The image is provided as a char buffer plus a unique ID. All management of the image stream source is in a separate thread.

When the UI thread requests the latest image from the image source it is provided with a pair comprising a shared_ptr containing the frame's char buffer and an integer ID. If the ID is different from the last frame on the QML side, the view is repainted with the new image. Otherwise it is ignored until the next timer trigger.

The QuickPaintedItem holds a QPixmap member. Each time a new image is received, this is filled, via convertFromImage, from a QImage constructed from the received buffer and then update() is called. A paint() method implemented in the class is responsible for drawing the updated pixmap.

The use of the shared_ptr makes copying the image between the threads fast while still addressing the lifetime issues. Obviously there is a lock on the request for the latest image but that is the only explicit synchronisation needed. While this frame is current, both sides will have a reference to it. Once a new one comes in on the stream manager side, it will be replaced with that and a reference dropped. The reference on the QML side will be dropped once it has finished updating its QPixmap member from it. Deletion of the held object in a shared_ptr is thread safe according the the standard.

[GrecKo]

@Wertyus expose a C++ property or an invokable function you can pass the VideoOutput videoSink property to from QML.

[Wertyus]

@JoeCFD I have worked extensively with GStreamer before, so I’m fairly confident (about 95%) that I can implement this solution if needed. However, I would like to explore other approaches first before switching to GStreamer or FFmpeg.

Currently, my goal is to make it work using Qt's built-in capabilities. If I run into further issues, I will definitely consider using GStreamer as a fallback solution.

Thanks again :) İf ı need your help about that, i will ask you :)

[Wertyus]

@Bob64

Thank you for sharing your approach! Using QQuickPaintedItem along with a timer and shared_ptr for thread-safe image handling sounds like an efficient way to manage the rendering process.

Right now, I am trying to make it work using Qt’s built-in solutions, but if I run into synchronization or performance issues, your approach could be a great alternative.

One thing I’m curious about:

Did you experience any latency issues with the QQuickPaintedItem approach when handling high frame rates (e.g., 30 FPS)?
Also, how did you handle buffer updates efficiently when new frames arrived?
Thanks again for the suggestion! I really appreciate it.

[Wertyus]

@GrecKo

Thanks for the suggestion! I actually tried exposing a C++ property and passing the videoSink to VideoOutput in QML. However, for some reason, QML does not recognize the VideoOutput element at all.

I checked that I have the required Qt Multimedia imports, but it still doesn’t work. Do you have any idea what might be causing this issue?

Could it be a missing dependency or something related to the QML engine initialization?
Do I need to manually register VideoOutput in my C++ code for it to be recognized?
I’d appreciate any insights you might have!

Thanks again for your help.

[GrecKo]

VideoOutput needs the Qt Multimedia module and importing it.
You don't need to manually register it.

Without code and logs we can't help you further.

[Bob64]

@Wertyus said in How to Display Frames in QML from C++ for Real-Time Applications:

Did you experience any latency issues with the QQuickPaintedItem approach when handling high frame rates (e.g., 30 FPS)?

I did some initial experiments to test the viability of the general approach and was able to get frame rates above 30fps. Since implementing the real thing, I have not explicitly measured this but it has been good enough for our purposes. In our case, the frames are provided not by a video stream but by a rendering of a 3D model provided by a back end server. Using QQuickPaintedItem was driven by our need to be able to react to mouse events and so on that are fed back to the server.

It might be that for your case, treating it as a video stream and using the specialised support for that would be more appropriate.

Also, how did you handle buffer updates efficiently when new frames arrived?

This is a sketch of the code in the QQuickPaintedItem. Because it's a "pull" approach, it is possible that frames could arrive too quickly for this to display all of them and anything between the last image requested and the current one will have been dropped.

void ViewerItem::checkForNewImage()
{
    // Called on timer trigger

    // Note no significant copying here.
    // `imageProvider` works on separate thread; `currentImage()` is mutex protected internally
    std::pair<std::shared_ptr<Image>, int> image = imageProvider->currentImage();

    if (image.second != m_currentIndex) {
        // New image - update the pixmap member
        m_pixmap.convertFromImage(
            QImage(image.first->pixelBuffer(),
                   image.first->width, image.first->height,
                   image.first->width*3, QImage::Format_RGB888
            )
       );
      //  New image so update index
      m_currentIndex = image.second;
      // update() is QQuickPaintedItem member - will call `paint(QPainter*)` on the present class
      // our implementation of  paint() uses `drawPixmap(0, 0, m_pixmap)`
      update()
   }
}

[charles453]

This post is deleted!