I'm also asking this on NXP forum, but I figured I'd try and get exposure from Qt side for a pressing issue: I need to be able to implement a streaming video player inside my Qt5.6 Qt app using QtQuick2 view. The proposed solution is to use QtMultimedia... the frontend is QML ( import QtMultimedia 5.0 , and use MediaPlayer {} and VideoOutput {} ) provided to supply an rtsp://url to an IP camera stream. The platform details are as follows: imx6q target with embedded display yocto-based embedded linux base wayland (not XCB, and not EGLFS) Qt 5.6 QtMultimedia configured to use gstreamer 1.0 OpenEmbedded Layer Index - qtmultimedia http://layers.openembedded.org/layerindex/recipe/49396/ gstreamer 1.0 plugins OpenEmbedded Layer Index - gstreamer1.0-plugins-imx http://layers.openembedded.org/layerindex/recipe/48930/ To get any form of reasonable performance I need to utilize these gstreamer imx plugins in the pipeline. My problem is, I must use QtMultimedia to integrate into the QML scenegraph, and it generates its own pipeline through very non-obvious means. I was able to hack QtMultimedia to display their 'playBin' pipeline contents shown here, but still doesn't show the videoSink at the end. (you can open image in its own tab to zoom in and view the details) qtmultimedia playbin gstreamer pipeline after running for a while I can't trace in the qtmultimedia source code how this video sink comes about, but after enabling debug logging from QtMultimedia it does mention "qgstvideorenderersink0" in a terse way, indicating it's a gstreamer element. qgstvideorenderersink doesn't appear anywhere in gst-inspect-1.0 so Qt is deriving this somehow else, but I don't know from where or how it behaves. It seems to be utilizing some hardware acceleration because the performance looks good while the video stream is playing. THE PROBLEM -- This pipeline 'works' for a limited amount of time, then fails badly. I absolutely need it to be able to recover cleanly. Firstly, periodically the pipeline is producing some h264 parsing errors: Error: "Error parsing H.264 stream" NOTE - every time this happens, we see the video flicker, then resume playing. I also begin noticing that various pipeline elements have new enumerators after them, as seen in the image above (e.g. rtpsession8, etc). If I showed a printout of the original pipeline, it shows 0 indexed instances of every plugin, so it's like we're jumping to new pipeline copies (and maybe not cleaning up properly). After several of these periodic H.264 errors over a period of say, up to 60 minutes, The application reaches an unrecoverable error: GLib-ERROR **: ../../glib-2.46.2/glib/gmem.c:100: failed to allocate 65611 bytes I suspect my memory allocation problem has something to do with so called 'zero-copy operations' not being respected by whatever videoSink QtMultimedia provides... This page is the only shred of information I can come up with. gstreamer-imx/zerocopy.md at master · Freescale/gstreamer-imx · GitHub https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md I produced a similar problem testing with a very old (third party) qt-gstreamer implementation which provides the plugin qtquick2videosink instead of qt's own inhouse solution. I could get the pipeline to run for a very long time with this one (no H.264 parsing errors that appear to reallocate a whole new pipeline automatically), but if I try to clean up the pipeline and assemble a different one myself with qtquick2videosink from qt-gstreamer (such as is needed to change the source to a different rtsp URI), I experience leaks which results in a similar memory allocation failure in imxvpudec plugin (CMA/DMA allocs). If I take the exact same pipeline but direct to imxeglvivsink or imxipuvideosink instead, I can allocate, then clean up the pipeline indefinitely with no issue of vpu running out of CMA/DMA alloc space. It's like it 'knows' how to clean up these CMA/DMA buffers properly only with imx provided video sinks. Here's a similar working gst-launch test that runs on my platform: gst-launch-1.0 rtspsrc location=rtsp://admin:admin@10.50.15.254:554/mpeg4cif name=source ! queue ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideosink NOTE - also works fine with imxeglvivsink I'd love to be able to just use imxeglvivsink OR imxipuvideosink directly, but I can't figure out how to get it to integrate into my Qt app scene correctly. I've tried forcing QtMultimedia to use this in place of its own weird sink by setting this environment variable: export QT_GSTREAMER_WINDOW_VIDEOSINK=imxeglvivsink I can see that Qt appears to recognize this variable in their source code, but the approach isn't working for me. It doesn't appear to solve the memory allocation problem, and the pipeline debug output still reports it's using qgstvideorenderersink0. Is there a way to actually use a supported video sink and have it work inside the Qt scene graph the way qgstvideorenderersink does? I also have an application where I build the pipeline up manually and direct to an imx videosink, it seems to blit to the screen ok the first time I tell it to play, but stops drawing to the screen when I try to interact at all with my Qt application. Is there a way to force imxeglvivsink 'on top' all the time in wayland? I think if I could do this I could get away with using the 'supported' imx sinks, and just have the video render over the top of the rest of the Qt app. So there you have it. Only when I introduce QtQuick2 wayland compatible video sinks do I somehow break with the zero-copy pipeline, and then I land in trouble because it leads to unrecoverable leaks. This is my theory, anyway. Even if it proves true, I'm not sure what the best approach to solving this is. I am in depserate need of help at this point, any and all is seriously appreciated! Thank you! AD

Hi, Since you need a bit more control over the pipeline, you should consider the QtGStreamer project . It provides syncs for QtQuick and a more fine grained control about building the pipeline.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

QtMultimedia gstreamer1.0 rtsp video + wayland + imx6 gst-imx-plugins - memory leaks?

Unsolved Mobile and Embedded

5 Posts 2 Posters 4.1k Views

A Offline
A Offline
adowdy
wrote on last edited by

#1
I'm also asking this on NXP forum, but I figured I'd try and get exposure from Qt side for a pressing issue:

I need to be able to implement a streaming video player inside my Qt5.6 Qt app using QtQuick2 view. The proposed solution is to use QtMultimedia... the frontend is QML ( import QtMultimedia 5.0 , and use MediaPlayer {} and VideoOutput {} ) provided to supply an rtsp://url to an IP camera stream.

The platform details are as follows:
imx6q target with embedded display
yocto-based embedded linux base
wayland (not XCB, and not EGLFS)
Qt 5.6
QtMultimedia configured to use gstreamer 1.0
OpenEmbedded Layer Index - qtmultimedia
http://layers.openembedded.org/layerindex/recipe/49396/
gstreamer 1.0 plugins
OpenEmbedded Layer Index - gstreamer1.0-plugins-imx
http://layers.openembedded.org/layerindex/recipe/48930/

To get any form of reasonable performance I need to utilize these gstreamer imx plugins in the pipeline.
My problem is, I must use QtMultimedia to integrate into the QML scenegraph, and it generates its own pipeline through very non-obvious means. I was able to hack QtMultimedia to display their 'playBin' pipeline contents shown here, but still doesn't show the videoSink at the end. (you can open image in its own tab to zoom in and view the details)

qtmultimedia playbin gstreamer pipeline after running for a while

I can't trace in the qtmultimedia source code how this video sink comes about, but after enabling debug logging from QtMultimedia it does mention "qgstvideorenderersink0" in a terse way, indicating it's a gstreamer element. qgstvideorenderersink doesn't appear anywhere in gst-inspect-1.0 so Qt is deriving this somehow else, but I don't know from where or how it behaves. It seems to be utilizing some hardware acceleration because the performance looks good while the video stream is playing.

THE PROBLEM --
This pipeline 'works' for a limited amount of time, then fails badly. I absolutely need it to be able to recover cleanly. Firstly, periodically the pipeline is producing some h264 parsing errors:
Error: "Error parsing H.264 stream"
NOTE - every time this happens, we see the video flicker, then resume playing. I also begin noticing that various pipeline elements have new enumerators after them, as seen in the image above (e.g. rtpsession8, etc). If I showed a printout of the original pipeline, it shows 0 indexed instances of every plugin, so it's like we're jumping to new pipeline copies (and maybe not cleaning up properly).

After several of these periodic H.264 errors over a period of say, up to 60 minutes, The application reaches an unrecoverable error:

GLib-ERROR **: ../../glib-2.46.2/glib/gmem.c:100: failed to allocate 65611 bytes

I suspect my memory allocation problem has something to do with so called 'zero-copy operations' not being respected by whatever videoSink QtMultimedia provides... This page is the only shred of information I can come up with.

gstreamer-imx/zerocopy.md at master · Freescale/gstreamer-imx · GitHub
https://github.com/Freescale/gstreamer-imx/blob/master/docs/zerocopy.md

I produced a similar problem testing with a very old (third party) qt-gstreamer implementation which provides the plugin qtquick2videosink instead of qt's own inhouse solution. I could get the pipeline to run for a very long time with this one (no H.264 parsing errors that appear to reallocate a whole new pipeline automatically), but if I try to clean up the pipeline and assemble a different one myself with qtquick2videosink from qt-gstreamer (such as is needed to change the source to a different rtsp URI), I experience leaks which results in a similar memory allocation failure in imxvpudec plugin (CMA/DMA allocs).

If I take the exact same pipeline but direct to imxeglvivsink or imxipuvideosink instead, I can allocate, then clean up the pipeline indefinitely with no issue of vpu running out of CMA/DMA alloc space. It's like it 'knows' how to clean up these CMA/DMA buffers properly only with imx provided video sinks.

Here's a similar working gst-launch test that runs on my platform:
gst-launch-1.0 rtspsrc location=rtsp://admin:admin@10.50.15.254:554/mpeg4cif name=source ! queue ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideosink
NOTE - also works fine with imxeglvivsink

I'd love to be able to just use imxeglvivsink OR imxipuvideosink directly, but I can't figure out how to get it to integrate into my Qt app scene correctly. I've tried forcing QtMultimedia to use this in place of its own weird sink by setting this environment variable:
export QT_GSTREAMER_WINDOW_VIDEOSINK=imxeglvivsink

I can see that Qt appears to recognize this variable in their source code, but the approach isn't working for me. It doesn't appear to solve the memory allocation problem, and the pipeline debug output still reports it's using qgstvideorenderersink0. Is there a way to actually use a supported video sink and have it work inside the Qt scene graph the way qgstvideorenderersink does?

I also have an application where I build the pipeline up manually and direct to an imx videosink, it seems to blit to the screen ok the first time I tell it to play, but stops drawing to the screen when I try to interact at all with my Qt application. Is there a way to force imxeglvivsink 'on top' all the time in wayland? I think if I could do this I could get away with using the 'supported' imx sinks, and just have the video render over the top of the rest of the Qt app.

So there you have it. Only when I introduce QtQuick2 wayland compatible video sinks do I somehow break with the zero-copy pipeline, and then I land in trouble because it leads to unrecoverable leaks. This is my theory, anyway. Even if it proves true, I'm not sure what the best approach to solving this is. I am in depserate need of help at this point, any and all is seriously appreciated! Thank you!
- AD
1 Reply Last reply
0
S Offline

S Offline
SGaist Lifetime Qt Champion
wrote on last edited by

#2

Hi,

Since you need a bit more control over the pipeline, you should consider the QtGStreamer project. It provides syncs for QtQuick and a more fine grained control about building the pipeline.
Interested in AI ? www.idiap.ch
Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

A 1 Reply Last reply
0
A Offline
A Offline
adowdy
replied to SGaist on last edited by

#3
@SGaist - Thank you for responding! -- let me clarify a bit. When QtMultimedia did not work the first thing I did was try QtGstreamer (synonymous with 'qt-gstreamer'). With more fine control of the pipeline I was indeed able to clear up H.264 parsing errors (so long as the pipeline was playing it would continue), but the video sink still doesn't handle VPU memory allocations cleanly when a state-change/restart of the pipeline is required.

In my original post I mentioned the approach with qt-gstreamer did not work due to a similar problem. I tried to keep it brief since my post was already getting so long... but the very same allocation problem seems to occur with their qtquick2videosink which is the only viable option for my QQuickView Qt5.6 application. On top of that, the qt-gstreamer project has not been maintaned for 4+ years. I'll also mention that performance becomes a problem using qt-gstreamer even in its limited working capacity, because QSGRenderThread CPU usage skyrockets and memory bandwidth is dramatically used up on even a 480p stream at 15 fps, which isn't good enough for a complex 'real-world' application.

QtMultimedia is the solution that 'replaces' third-party stopgaps such as 'qt-gstreamer' and I need it to work on imx6 wayland -- so I'm definitely still on the hunt for the solution to the core QtMultimedia problem.

Where can I make the bug report on QtMultimedia where it can get Qt developer attention? I get the feeling that using Wayland Qt5 on an imx6 is a common enough usecase to be concerned.
- AD
1 Reply Last reply
0
S Offline

S Offline
SGaist Lifetime Qt Champion
wrote on last edited by

#4

My bad, I managed to overlook that paragraph.

Even if long, it's detailed which is good.

The bug report system is a starting point. You can also reach Qt's developers on the interest mailing list (This forum is more user oriented)

By the way, where did you clone QtGstreamer from ?

There are (not a lot but still) commits from last year and even this year.
Interested in AI ? www.idiap.ch
Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

A 1 Reply Last reply
0
A Offline
A Offline
adowdy
replied to SGaist on last edited by

#5
@SGaist - thank you for providing more info.

To answer your question, I am pulling QtGstreamer version 1.2.0 from a tarball here:

https://gstreamer.freedesktop.org/src/qt-gstreamer/qt-gstreamer-1.2.0.tar.xz

there are some patches applied afterward, but this archive was published in 2014.

To configure and build, I am utilizing this fellow's meta-qt5-extra Yocto layer

https://github.com/schnitzeltony/meta-qt5-extra

The qt-gstreamer patches applied do not appear to correlate with any of the behavior I described, so for awhile I was trying to see if anyone got further in a similar platform case with qt-multimedia. Turning up empty handed so far on this solution.

I should mention for the record, I got someone on NXP forum to send some patches that potentially improve things QtMultimedia side.

Links:

https://github.com/kernevil/qtmultimedia/commit/198835aa7c9e25e724fa9a80594ff8a7d7bb02ab
gstreamer-imx plugins side:
https://github.com/kernevil/gstreamer-imx/commit/47a56f0d49b13f605cb55f19c0ec515cb91d96c6
the above commit seems to be merged into latest freescale gstreamer-imx at v 0.13.0 as well:
https://github.com/Freescale/gstreamer-imx/commit/47a56f0d49b13f605cb55f19c0ec515cb91d96c6

I am still testing these now, no luck yet after applying the qtmultimedia patch and testing with gstreamer-imx version 0.13.0, but I am exploring this in more detail now.

At some point soon I will also try to follow up with the Qt channels you mentioned. Thanks again!
- AD
1 Reply Last reply
0