Simple app to show webcam output via GStreamer pipeline

HanSooloo

I am trying to build a simple app on macOS that can use a GStreamer pipeline to open the webcam as a video source and output that to the Qt app.

I have looked at a few examples to draw inspiration:

https://doc.qt.io/qtforpython-6/examples/example_multimedia_player.html
https://doc.qt.io/qt-5/qmediaplayer.html#setMedia (specifically the gst-pipeline example)

I have the following program, but the output is a black content inside the app window .. this is different from a "blank" Qt app that has a brownish background that seems to be default theme color. How can I get my program to output the webcam video to the Qt app?

One interesting data point is that macOS does NOT show the "camera in use" icon in the toolbar with this program.

Black background with example code:

Default app with nothing inside:

import sys

from PySide6.QtWidgets import QApplication, QMainWindow, QPushButton, QWidget
from PySide6.QtCore import QSize, Qt, QUrl, Slot
from PySide6.QtMultimedia import QMediaPlayer
from PySide6.QtMultimediaWidgets import QVideoWidget

GST_PIPELINE = ('avfvideosrc device-index=1 ! '
                '"video/x-raw, width=1280, height=720, format=(string)YUY2, texture-target=rectangle" ! '
                'rawvideoparse width=1280 height=720 format=yuy2 ! '
                'queue ! '
                'autovideoconvert ! '
                'qtvideosink')

class MainWindow(QMainWindow):
  def __init__(self):
    super().__init__()
    self._player = QMediaPlayer()
    self._video_widget = QVideoWidget()
    self.setCentralWidget(self._video_widget)
    self._player.setVideoOutput(self._video_widget)
  
  def closeEvent(self, event):
    self._ensure_stopped()
    event.accept()
    
  @Slot()
  def _ensure_stopped(self):
      if self._player.playbackState() != QMediaPlayer.StoppedState:
          self._player.stop()

  def _play(self):
    self._player.setSource(QUrl(f'gst-pipeline: {GST_PIPELINE}'))
    self._player.play()


if __name__ == '__main__':
  app = QApplication(sys.argv)
  window = MainWindow()
  window.show()
  window._play()
  sys.exit(app.exec())

JoeCFD

@HanSooloo
FFmpeg as the default backend from here:
https://doc.qt.io/qt-6/qtmultimedia-index.html

You need to change backend to gstreaner to run your pipeline
export QT_MEDIA_BACKEND=gstreamer

On iOS, I am not sure.

Check if the qml6glsink is available since it may not be built in the gstreamer installed.
run command:
gst-inspect-1.0 | grep qml
to see if qml6 sink is available

If yes, do
export GST_DEBUG=5
to see what is wrong with gstreamer

HanSooloo

@JoeCFD said in Simple app to show webcam output via GStreamer pipeline:

@HanSooloo
FFmpeg as the default backend from here:
https://doc.qt.io/qt-6/qtmultimedia-index.html

You need to change backend to gstreaner to run your pipeline
export QT_MEDIA_BACKEND=gstreamer

On iOS, I am not sure.

Check if the qml6glsink is available since it may not be built in the gstreamer installed.
run command:
gst-inspect-1.0 | grep qml
to see if qml6 sink is available

If yes, do
export GST_DEBUG=5
to see what is wrong with gstreamer

gst-inspect-1.0 doesn't show anything related to qml ... Do I need to install GStreamer with QML support then?

Is there a documented way to do this via Homebrew on macOS?

JoeCFD

@HanSooloo install gstreamer good plugin which has qml gl sink. Unluckily, there is not much doc for you to refer to.

HanSooloo

@JoeCFD I configured the GStreamer source using the meson configure -Dqt6=enabled builddir configuration option and built with regular ninja command, and finally installed via meson install.

I was able to confirm Qt6 support with the following:

$ gst-inspect-1.0 | grep qml
qml6:  qml6glmixer: Qt6 Video Mixer
qml6:  qml6gloverlay: Qt Video Overlay
qml6:  qml6glsink: Qt6 Video Sink
qml6:  qml6glsrc: Qt Video Source

After updating my pipeline as follows, I am still getting the black screen in the window. Any thoughts on where to go from here?

GST_PIPELINE = ('avfvideosrc device-index=1 ! '
                '"video/x-raw, width=1280, height=720, format=(string)YUY2, texture-target=rectangle" ! '
                'rawvideoparse width=1280 height=720 format=yuy2 ! '
                'queue ! '
                'autovideoconvert ! '
                'qml6glsink')

JoeCFD

@HanSooloo Does your pipeline run from command line? Wait, Qt6 may have removed support for running gstreamer pipeline in Qt. You may need to find it out. If my memory serves me right, Qt5 supports it. I do not use QMultimedia module for video display. Instead, I use raw gstreamer code to do it.

You can check this example out:
https://github.com/GStreamer/gst-plugins-good/tree/master/tests/examples/qt/qmlsink
BTW, QMultimedia module does not offer enough controls over video play.

HanSooloo

@JoeCFD said in Simple app to show webcam output via GStreamer pipeline:

@HanSooloo Does your pipeline run from command line? Wait, Qt6 may have removed support for running gstreamer pipeline in Qt. You may need to find it out. If my memory serves me right, Qt5 supports it. I do not use QMultimedia module for video display. Instead, I use raw gstreamer code to do it.

You can check this example out:
https://github.com/GStreamer/gst-plugins-good/tree/master/tests/examples/qt/qmlsink
BTW, QMultimedia module does not offer enough controls over video play.

The pipeline with autovideosink works on the command line. However, the one with qml6glsink doesn't work, since it is expecting a QGuiApplication context, e.g.,

(pipeline modified to a super simple video test pattern for brevity)

$ gst-launch-1.0 -v -e  videotestsrc ! autovideoconvert ! qml6glsink
Setting pipeline to PAUSED ...
ERROR: from element /GstPipeline:pipeline0/GstQml6GLSink:qml6glsink0: Failed to connect to Qt
Additional debug info:
../subprojects/gst-plugins-good/ext/qt6/gstqml6glsink.cc(353): gst_qml6_gl_sink_change_state (): /GstPipeline:pipeline0/GstQml6GLSink:qml6glsink0:
Could not retrieve QGuiApplication instance
ERROR: pipeline doesn't want to preroll.
Failed to set pipeline to PAUSED.
Setting pipeline to NULL ...
Freeing pipeline ...

JoeCFD

@HanSooloo you can not do it. From command line, gstreamer has nothing to do with Qt.
gst-launch-1.0 -v -e videotestsrc ! autovideoconvert ! autovideosink
qml6glsink is attached to a qt widget or qml item. Check the example out.

HanSooloo

@JoeCFD as you suggested, understand that Qt6 with GStreamer backend on macOS is not an option.

So, decided to switch back to Qt5 just to validate a basic GStreamer pipeline. Even with the below code that uses a simple video test source, I am still getting a black window content.

Any thoughts on where to go from here?

import sys

from PyQt5.QtWidgets import QApplication, QMainWindow, QPushButton, QWidget
from PyQt5.QtCore import QSize, Qt, QUrl
from PyQt5.QtMultimedia import QMediaPlayer, QMediaContent
from PyQt5.QtMultimediaWidgets import QVideoWidget

GST_PIPELINE = 'videotestsrc ! autovideoconvert ! autovideosink'


class MainWindow(QMainWindow):
  def __init__(self):
    super().__init__()
    self._player = QMediaPlayer()
    self._video_widget = QVideoWidget()
    self.setCentralWidget(self._video_widget)
    self._player.setVideoOutput(self._video_widget)

  def _play(self):
    self._player.setMedia(QMediaContent(QUrl(f'gst-pipeline: {GST_PIPELINE}')))
    self._player.play()


if __name__ == '__main__':
  app = QApplication(sys.argv)

  window = MainWindow()
  window.show()
  window._play()

  sys.exit(app.exec())

JoeCFD

@HanSooloo https://forum.qt.io/topic/141301/understanding-qt-documentation-about-using-qtmultimedia-with-lgpl-gstreamer-on-android/8

the sink is qtvideosink

malikcis

@HanSooloo said in Simple app to show webcam output via GStreamer pipeline:

qml6glsrc

Hi,
Any idea on how to use qml6glsrc?
Since QT6 Multimedia does not support gstreamer I don't know how qml6glsrc is supposed to work.

JoeCFD

@malikcis some made an example here.
https://forum.qt.io/topic/155422/parallel-pipelines-with-qml6glsink?_=1712071475549

JoeCFD

@JoeCFD gstreamer 1.24 build includes qml6glsink automatically if qt6 is installed in Ubuntu. No extra settings is needed in the build of gstreamer 1.24.