Seamless video-to-video transition with v4l2h264dec + waylandsink on Pi3 (Debian Trixie) — frozen frame between clips

The Qt Mayssa

Platform: Raspberry Pi 3B, Debian Trixie, Qt 5.15, GStreamer 1.24, Wayland compositor, v4l2h264dec

Why we built a custom GStreamer item instead of using Qt Multimedia's Video element

Before describing the transition problem itself, I want to explain why we're not using Qt Multimedia's Video QML element, since that's the obvious first question.

Qt Multimedia's GStreamer backend does use v4l2h264dec — we confirmed this with GST_DEBUG=3. That's not the issue. The issue is what happens at EOS on this specific platform.

On Pi3 / kernel 6.6, when GST_EVENT_EOS reaches the V4L2 sink, it triggers a known memory leak in the Pi's V4L2 driver at end-of-playback. For a digital signage player looping content 24/7, this CMA (Contiguous Memory Allocator) leak accumulates until V4L2 buffer allocation fails:

v4l2h264dec: Failed to allocate output buffers — Cannot allocate memory

The workaround is to intercept and drop GST_EVENT_EOS via a pad probe on the sink pad, before it reaches the driver. End-of-video is detected via GST_MESSAGE_EOS on the bus (which is still emitted normally).

Qt Multimedia's Video element doesn't expose any hook to install a pad probe at this level. This is why we built GstVideoItemV2, a custom QQuickItem that drives the pipeline directly with the raw GStreamer C API.

Our pipeline:

filesrc → qtdemux → h264parse → v4l2h264dec → queue → waylandsink
                                                             ↑
                                               pad probe: drop GST_EVENT_EOS

Note on Qt6:

migrating to Qt6 is not on our roadmap today, but if Qt6 Multimedia's GStreamer backend exposes pad-probe-level hooks, or if newer kernels have fixed this V4L2 EOS leak, it would be a strong argument to prioritize the upgrade. Any feedback on Qt6 + Pi3 + Wayland experience is welcome.
The problem: ~13–80ms frozen frame at every video transition
In a playlist zone, our player pre-builds the next video's pipeline while the current video is still playing (buildPipeline() runs on a separate dedicated worker thread). When video 1 ends:

Video 1:  PLAYING ──────────────────────── STATE_NULL (~9ms, holds v4l2 lock)
Video 2:     buildPipeline (11ms, no lock) ──────── waits lock ── set_state(PLAYING)
Screen:   video 1 last frame ─────────────────────── FROZEN ─────── video 2 starts

The freeze is not a black screen — it's video 1's last frame held by the Wayland compositor while video 2's waylandsink hasn't rendered its first frame yet.

Loop works fine: when the same video loops (seek to 0), we reuse the same pipeline and the same waylandsink — no teardown, no lock, no freeze. The problem is strictly video1 → video2.

Why we can't keep waylandsink alive between videos
Reusing waylandsink across different source files (to avoid teardown) causes additional CMA exhaustion after ~10–15 cycles on this kernel. The V4L2 decoder's DMA buffers are tied to the sink's Wayland buffer pool and aren't released until STATE_NULL. We must destroy and recreate the full pipeline on every transition.

One hardware decoder, one at a time
The Pi3 has one V4L2 H.264 decoder instance (/dev/video10). We serialize access with a static QMutex that gates both the STATE_NULL release (video 1's worker thread) and the set_state(PLAYING) acquisition (video 2's worker thread).

Current mitigation: worker thread pre-warming
Each GstVideoItemV2 runs all GStreamer ops on a dedicated QThread worker. On Pi3 ARMv7 @ 1.4 GHz, pthread_create → kernel stack allocation → first event loop dispatch takes ~40–50ms. We pre-create the next video's QQuickItem (and its worker thread) during playback of the current video. Best-case gap: ~13ms. Still perceptible.

Questions

Atomic waylandsink handover: is there a GStreamer pattern to swap the source decoded into a single persistent sink without going through STATE_NULL — on a system where the sink can't be kept alive across sources due to CMA constraints?
input-selector: has anyone used it upstream of a shared sink with two independent decoder chains on Pi3 / kernel 6.6, managing the CMA budget?
kmssink + DRM atomic flip: viable on Pi3 + Trixie to bypass compositor buffering and hide the handover gap?
STATE_PAUSED vs STATE_NULL: does PAUSED release V4L2 CMA buffers on this kernel? If it holds the hardware open (preventing dual-open) but releases CMA, we could keep a warm pipeline.
Qt6 Multimedia: does it handle the V4L2 EOS CMA leak differently on Pi3/Wayland? Any first-hand experience?
Thanks in advance for any pointers.

SGaist

Hi,

I can only answer on the last part: Qt6 Multimedia module currently uses ffmpeg as backend on all platform except for embedded Linux which might be your case. I suggest you take a look at GStreamer chapter in the Qt Multimedia documentation. There might be stuff in there that could be of interest.

Hope it helps

The Qt Mayssa

Hey ! @SGaist & everyOne else reading this !
[UPDATE]
Since posting this, we have identified and fixed two additional issues on the same platform. Sharing here since they may be relevant to others hitting similar symptoms.

1. CMA leak at EOS — root cause confirmed, workaround validated

The EOS CMA leak described above (V4L2 buffer allocation failure after ~10–15 cycles) is confirmed on Pi3 / kernel 6.6. Our pad probe dropping GST_EVENT_EOS before it reaches the V4L2 sink remains the only reliable workaround. No regression observed after hundreds of cycles.

2. New: fallback buffer pool leak in v4l2h264dec (~6 MB per session)

Independently of the EOS/CMA issue, we discovered a second leak: v4l2h264dec leaks ~6 MB of virtual memory per decode session when downstream does not respond to GST_QUERY_ALLOCATION. Exactly 2 frames (2 × ~3.1 MB) are never freed on pipeline teardown.

Root cause: the decoder logs didn't get downstream ALLOCATION hints → no pool, making new pool and creates a fallback copy pool whose buffers are not fully released at STATE_NULL. Neither fakesink nor Qt’s qgstvideorenderersink respond to this query — both reproduce the leak.

Fix: intercept the ALLOCATION query on v4l2h264dec’s src pad and answer with GstVideoMeta support before it reaches downstream. This eliminates the fallback pool entirely — measured at 0.00 MB/session on a 300-run loop.

We filed a separate GStreamer bug for this: [link to new ticket gstreamer] , & new qt forum ticket

Status on original questions:

élément de listeAtomic waylandsink handover / input-selector: not pursued — we moved to software rotation (QML-level), which removed the need to manage multiple waylandsink instances simultaneously.
élément de listeSTATE_PAUSED vs STATE_NULL: PAUSED does not release V4L2 CMA buffers on this kernel — confirmed. Full STATE_NULL remains required on every transition.
élément de listeQt6: not tested, still not on our roadmap.

SGaist

@The-Qt-Mayssa thanks fo the detailed update !

It looks like you did not post the link for the GStreamer ticket.

The Qt Mayssa

@SGaist my bad ! should be good now ^^ .
Have a good day

SGaist

Yes, it is all good. Thanks !

Same to you