QT6 qtwayland: Problem with gstreamer, waylandsink a QWidget not showing video stream

Terry Barnaby

Hi, We are testing the use of QT6 on an NXP IMX8mp Yocto system. This has QT6, wayland, gstreamer and Weston as the GUI packages.

We have used QT5 in the past and managed to get a gstreamer pipeline to display Video within a QWidget by sending the QWidgets Wayland surface to waylandsink. This worked reasonably well and was efficient.

We are truny the same with QT6 on the latest NXP Yocto release. If the QWidget is a top level widget, created with QWidget(0), this works fine. However if the QWidget is further down the QWidget hierarchy the gstreamer pipeline stalls and no video is shown (no errors). It appears that the QWidget's surface is somehow inactive and the surface is not re-drawn.
The QWidget is created with: setAttribute(Qt::WA_NativeWindow); to make sure it has a native Wayland surface and use is made of QPlatformNativeInterface to get the Wayland surface for the QWidget.

I have also tested using QT6 on Fedora37 with a Plasma/Wayland session and this has the same issue.
Any ideas on what may be happening, I think it may be a qtwayland issue ?

JoeCFD

@Terry-Barnaby Just curious: how do you display video within a qwidget with a gstreamer pipeline?

Terry Barnaby

@JoeCFD

I use a technique that is mentioned on various Internet sites and in gstreamer. This worked well with QT5. Roughly you create the QWidget with Qt::WA_NativeWindow to make sure it has its own Wayland surface.

You then have a gstreamer pipeline like: "videotestsrc is-live=1 ! video/x-raw,width=640,height=480,framerate=25/1 ! waylandsink name="videoSink""

Then when the waylandsink asks via a gstreamer bus request you use:

QPlatformNativeInterface* native = QGuiApplication::platformNativeInterface();
struct wl_display* display_handle = (struct wl_display*)native->nativeResourceForWindow("display", NULL);
GstContext* context;

context = gst_wl_display_handle_context_new(display_handle);
gst_element_set_context(GST_ELEMENT(GST_MESSAGE_SRC (message)), context);

then

struct wl_surface* surface = static_cast<struct wl_surface*>(native->nativeResourceForWindow("surface", widget->windowHandle()));
gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(GST_MESSAGE_SRC(message)), (guintptr)surface);
gst_video_overlay_set_render_rectangle(GST_VIDEO_OVERLAY(GST_MESSAGE_SRC(message)), 0, 0, widget->width(), widget->height());

This tells Wayland sink to use the QWidget's surface as its output destination rather than its own surface. On QWidget resize you can tell waylandsink about this.
It works very well using hardware colour conversion and scaling on an Imx8 and on Intel x86 systems.
I can let you have my test case source code if you like.

JoeCFD

@Terry-Barnaby I guess it is the similar approach as one of the two in QtGStreamer.
it would be nice to see your test code. I use another approach in QtGstreamer
and gstreamer example is here.
https://github.com/GStreamer/gst-examples/blob/master/playback/player/qt/main.qml

Maybe you try to overlay the video widget on the top of all other widgets and then it will work. And it will not affect anything.

Terry Barnaby

@JoeCFD
I Have put my basic test code in: https://portal.beam.ltd.uk/files//test011-qt6-video.tar.gz

This is cut-down and rough and ready. It has #defines for Fedora37 X11, Fedora37 Wayland and iMX8mp Yocto but that has specific tools directories in the Makefile.

When UseQWidgetTop is set to 1, it works with Fedora37 X11, Fedora37 Wayland and iMX8mp Yocto .

When UseQWidgetTop is set to 0 it only works with Fedora37 X11.
There is a very basic appsink in there, but that assumes hardcode video format and size.
I will have a look at the techniques you mention.

Terry Barnaby

@Terry-Barnaby
Ah just looked, the code you mention uses QML. My embedded system does not support that and really I don't want to add even more complexity and size to an already over complex system !

JoeCFD

@Terry-Barnaby Actually it is not so complicated as you think. I added only one qml sink with QQuickWidget in my widgets app. That is it.

JoeCFD

@Terry-Barnaby Thanks. But your file is not available.
Uri: files//test011-qt6-video.tar.gz not found

Terry Barnaby

@JoeCFD
Sorry it was in an Intranet location. Try here: https://portal.beam.ltd.uk/public/test011-qt6-video.tar.gz

On the QML its not the complexity of the User code, its the complexity underneath.

JoeCFD

@Terry-Barnaby Got it. Thanks a lot.

Terry Barnaby

Does anyone know if Qt::WA_NativeWindow actually works with Qt 6.5 ?

From my current debugging at the Wayland protocol level it doesn't appear to be working as I would expect. I am creating two QWidgets, one a child of the first top level QWidget. Both have the Qt::WA_NativeWindow flag set. I use QPainter to draw into each QWidget on QTimers. When I do this:

The QPlatformNativeInterface nativeResourceForWindow("surface", windowHandle()) function does return different pointers (wl_surface ??).
The Wayland protocol does appear to be creating multiple surfaces.
The Wayland drawing including the wl_surface@17.commit() for the two QWidget paints use the same Walyland surface ??
Using the Weston --debug option in conjunction with "weston-debug scene-graph" only shows the one surface in use by the application.

It seems like the qtwayland has somehow forced all drawing/interaction to the single native Wayland surface ?