QAbstractVideoFilter: enforce pixel format coming from GStreamer

adrianf0

I would like to process data coming from the camera and display the resulting frame. For that, I use QML Camera, VideoOutput and QAbstractVideoFilter. The image is properly displayed (in greyscale).

The camera's native pixel format is Y16 and the processing may take place only on top of that format. The filter would eventually generate and return QVideoFrame supported by the QVideoSurfaceFormat, however, for processing I need unmodified native Y16 data. I could also use shaders of VideoOutput to perform the final conversion to the format supported by the VideoOutput.

In QVideoFilterRunnable::run the input->pixelFormat is set to Format_RGB32. As far, as I understand, the conversion happens in the GStreamer backend. I guess, that GStreamer converts to the closest format indicated by QVideoSurfaceFormat of the VideoOutput. How could I interfere with this format negotiation process to enforce the native one? I think I should reimplement QAbstractVideoSurface::supportedPixelFormats() in the QML Camera, but I don't know how to access it (and if it's even that mechanism which is implemented in the QML Camera).

Following this hint, I can only get QCamera object of the Camera QML instance.

The solution should involve the minimum number of data copying due to performance reasons.

SGaist

Hi,

Is it a special camera ?
For GStreamer, you could maybe test a custom pipeline ?

adrianf0

Hi @SGaist ,
Yes, it's a custom camera.

It's not a problem of GStreamer, I can output the native format:

 gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw,format=GRAY16_LE,width=32,height=60' ! fakesink

The problem is QT requesting from GStreamer its preferred pixel format (I believe RGB32), thus GStreamer performs format conversion. How to avoid that?

SGaist

One way could be to use a custom pipeline as show in QMediaPlayer::setMedia.

adrianf0

Nice idea!
I have changed my qml to use MediaPlayer:

    MediaPlayer {
        id: falcon
        source: "gst-pipeline: v4l2src device=/dev/video0 ! video/x-raw,format=GRAY16_LE,width=32,height=60 ! qtvideosink"
    }

However, I get the error:

Error: " v4l2src device=/dev/video0 ! video/x-raw,format=GRAY16_LE,width=32,height=60 ! qtvideosink" : "could not link v4l2src0 to qgstvideorenderersink1, qgstvideorenderersink1 can't handle caps video/x-raw, format=(string)GRAY16_LE, width=(int)32, height=(int)60"

SGaist

How do you determine that ?

adrianf0

@SGaist 97?

In the GStreamer logs I find:

0:00:02.332433863  4254 0xaaaae219be00 INFO        GST_ELEMENT_PADS gstelement.c:928:gst_element_get_static_pad: found pad qgstvideorenderersink1:sink
0:00:02.332482614  4254 0xaaaae219be00 INFO                GST_PADS gstutils.c:1587:prepare_link_maybe_ghosting: capsfilter0 and qgstvideorenderersink1 in same bin, no need for ghost pads
0:00:02.665844297  4254 0xaaaae219be00 INFO                GST_PADS gstpad.c:2377:gst_pad_link_prepare: trying to link capsfilter0:src and qgstvideorenderersink1:sink
0:00:02.665934268  4254 0xaaaae219be00 INFO                GST_PADS gstpad.c:2434:gst_pad_link_prepare: caps are incompatible
0:00:02.665977289  4254 0xaaaae219be00 INFO                GST_PADS gstpad.c:2527:gst_pad_link_full: link between capsfilter0:src and qgstvideorenderersink1:sink failed: no common format
0:00:02.666031409  4254 0xaaaae219be00 INFO                 default gstutils.c:2149:gst_element_link_pads_filtered: Could not link pads: capsfilter:src - qgstvideorenderersink1:(null)
0:00:02.666068850  4254 0xaaaae219be00 INFO              GST_STATES gstelement.c:2688:gst_element_continue_state:<capsfilter0> completed state change to NULL

So apparently it fails, because VideOutput has different (RGB32) format.

I wanted to modify format from Y16 to RGB32 with VideoOutput shader or filter, but I can't as the link to VideoOutput can't be even initialized... Any idea how I could address that?

SGaist

@adrianf0 said in QAbstractVideoFilter: enforce pixel format coming from GStreamer:

@SGaist 97?

Strange typo that has been fixed.

VideoOutput handles more than just that format however, IIRC, it does support Y16 out of the box. One possible way is to modify QtMultimedia to add support for it or write your own camera backend.

adrianf0

You are right, VideoOutput doesn't support Y16 currently.
I solved the problem by writting a GStreamer plugin repacking unmodified Y16 pixel data to RGBx pixels data supported by the QML VideoOutput.

SGaist

Would you mind sharing the code ?

Additional formats can be supported in QtQuick, it requires a shader that converts the data.