Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

QAbstractVideoFilter: enforce pixel format coming from GStreamer



  • I would like to process data coming from the camera and display the resulting frame. For that, I use QML Camera, VideoOutput and QAbstractVideoFilter. The image is properly displayed (in greyscale).

    The camera's native pixel format is Y16 and the processing may take place only on top of that format. The filter would eventually generate and return QVideoFrame supported by the QVideoSurfaceFormat, however, for processing I need unmodified native Y16 data. I could also use shaders of VideoOutput to perform the final conversion to the format supported by the VideoOutput.

    In QVideoFilterRunnable::run the input->pixelFormat is set to Format_RGB32. As far, as I understand, the conversion happens in the GStreamer backend. I guess, that GStreamer converts to the closest format indicated by QVideoSurfaceFormat of the VideoOutput. How could I interfere with this format negotiation process to enforce the native one? I think I should reimplement QAbstractVideoSurface::supportedPixelFormats() in the QML Camera, but I don't know how to access it (and if it's even that mechanism which is implemented in the QML Camera).

    Following this hint, I can only get QCamera object of the Camera QML instance.

    The solution should involve the minimum number of data copying due to performance reasons.


  • Lifetime Qt Champion

    Hi,

    Is it a special camera ?
    For GStreamer, you could maybe test a custom pipeline ?



  • Hi @SGaist ,
    Yes, it's a custom camera.

    It's not a problem of GStreamer, I can output the native format:

     gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw,format=GRAY16_LE,width=32,height=60' ! fakesink
    

    The problem is QT requesting from GStreamer its preferred pixel format (I believe RGB32), thus GStreamer performs format conversion. How to avoid that?


  • Lifetime Qt Champion

    One way could be to use a custom pipeline as show in QMediaPlayer::setMedia.



  • Nice idea!
    I have changed my qml to use MediaPlayer:

        MediaPlayer {
            id: falcon
            source: "gst-pipeline: v4l2src device=/dev/video0 ! video/x-raw,format=GRAY16_LE,width=32,height=60 ! qtvideosink"
        }
    

    However, I get the error:

    Error: " v4l2src device=/dev/video0 ! video/x-raw,format=GRAY16_LE,width=32,height=60 ! qtvideosink" : "could not link v4l2src0 to qgstvideorenderersink1, qgstvideorenderersink1 can't handle caps video/x-raw, format=(string)GRAY16_LE, width=(int)32, height=(int)60"
    

  • Lifetime Qt Champion

    How do you determine that ?



  • @SGaist 97?

    In the GStreamer logs I find:

    0:00:02.332433863  4254 0xaaaae219be00 INFO        GST_ELEMENT_PADS gstelement.c:928:gst_element_get_static_pad: found pad qgstvideorenderersink1:sink
    0:00:02.332482614  4254 0xaaaae219be00 INFO                GST_PADS gstutils.c:1587:prepare_link_maybe_ghosting: capsfilter0 and qgstvideorenderersink1 in same bin, no need for ghost pads
    0:00:02.665844297  4254 0xaaaae219be00 INFO                GST_PADS gstpad.c:2377:gst_pad_link_prepare: trying to link capsfilter0:src and qgstvideorenderersink1:sink
    0:00:02.665934268  4254 0xaaaae219be00 INFO                GST_PADS gstpad.c:2434:gst_pad_link_prepare: caps are incompatible
    0:00:02.665977289  4254 0xaaaae219be00 INFO                GST_PADS gstpad.c:2527:gst_pad_link_full: link between capsfilter0:src and qgstvideorenderersink1:sink failed: no common format
    0:00:02.666031409  4254 0xaaaae219be00 INFO                 default gstutils.c:2149:gst_element_link_pads_filtered: Could not link pads: capsfilter:src - qgstvideorenderersink1:(null)
    0:00:02.666068850  4254 0xaaaae219be00 INFO              GST_STATES gstelement.c:2688:gst_element_continue_state:<capsfilter0> completed state change to NULL
    

    So apparently it fails, because VideOutput has different (RGB32) format.

    I wanted to modify format from Y16 to RGB32 with VideoOutput shader or filter, but I can't as the link to VideoOutput can't be even initialized... Any idea how I could address that?


  • Lifetime Qt Champion

    @adrianf0 said in QAbstractVideoFilter: enforce pixel format coming from GStreamer:

    @SGaist 97?

    Strange typo that has been fixed.

    VideoOutput handles more than just that format however, IIRC, it does support Y16 out of the box. One possible way is to modify QtMultimedia to add support for it or write your own camera backend.


Log in to reply