video doesn't display on LCD screen

[Jeffff]

@SGaist Very nice to get your reply:)
I can use gstreamer to show video on screen. However, due to UVC camera compatible reason, I cannot bridge camera with LCD directly with live display(by gst command). But I can output videos by using command(gst-launch-1.0 videotestsrc pattern=0 ! imxg2dvideosink). As far as I know, NXP has prepared some gstreamer plugin including camera, encoder and display, and my gstreamer are using these plugin(like imxg2dvideosink). I don't know whether my QT is adopting qst, and how QT works with gstreamer to display video, as well as what should I do to check or debug this part.
In my side, I've written my own code based on v4l2, which can output data(checked by converting data into JPG file), the image can also be converted to QImage format, which is the input of member function "VideoProducer::updateFrame" shown above

[SGaist]

Check with GStreamer's qmlsink qmlglsink, it might make your life easier.

[Jeffff]

@SGaist Can you introduce more detail about that? I'm quite a newbie in QT and don't get your point. Do you mean the QT Multimedia in imx6 must be using gstreamer for certain video processing(I guess such as scaling), and inside gstreamer, the plugin qmlsink if finally used for displaying? I searched the whole file system and I cannot get any file named with qmlsink. Also I searched my qtmultimedia source code by content, there is nothing found.
Meanwhile I find that there are 2 version of libgst existing in my target board, one is version 1.0, another is version0.1. I deleted all lib files of version 0.1, but nothing happens, the video still cannot be displayed on screen.

[SGaist]

Sorry, the name of the sink was wrong it's qmlglsink.

The QtMultimedia backend for Linux uses GStreamer, hower it has used it way before the qmlglsink existed and also it supports both the widgets and quick modules.

That said, it has no idea of any accelerated plugins like NXP is providing, hence using a custom pipeline is likely the best way to achieve what you want to do. The qmlglsink is part of the good plugin package of GStreamer however I don't remember in which version it has been made available so you should check that.

[Jeffff]

Thank you @SGaist .
Then I try to search my QtMultimedia by objdump:

It seems that this QtMultimedia doesn't use gstreamer. Also I searched my directory, no "qmlglsink" was found.
The QT version 5.10.1 is installed on this board. I really lost my direction.

[SGaist]

You are mixing things here.

QtMultimedia does not link to GStreamer. Its Linux plugin will be.
qmlglsink is part of GStreamer, but I don't remember since which version and it's part of one of the plugin packages of GStreamer.

[Jeffff]

I havn't found the "qmlglsink" plugin in my gstreamer1.0. Do you mean it is a must to use this plugin if we want to play video in QT?

-sh-4.4# gst-inspect-1.0 | grep sink
imxipu:  imxipuvideosink: Freescale IPU video sink
tcp:  tcpclientsink: TCP client sink
tcp:  tcpserversink: TCP server sink
tcp:  multifdsink: Multi filedescriptor sink
tcp:  multisocketsink: Multi socket sink
rtspclientsink:  rtspclientsink: RTSP RECORD client
imxv4l2video:  imxv4l2videosink: V4L2 CSI Video Sink
imxpxp:  imxpxpvideosink: Freescale PxP video sink
alsa:  alsasink: Audio sink (ALSA)
imxg2d:  imxg2dvideosink: Freescale G2D video sink
soup:  souphttpclientsink: HTTP client sink
udp:  udpsink: UDP packet sender
udp:  multiudpsink: UDP packet sender
udp:  dynudpsink: UDP packet sender
ximagesink:  ximagesink: Video sink
debug:  testsink: Test plugin
gio:  giosink: GIO sink
gio:  giostreamsink: GIO stream sink
playback:  playsink: Player Sink
autodetect:  autovideosink: Auto video sink
autodetect:  autoaudiosink: Auto audio sink
imxeglvivsink:  imxeglvivsink: Freescale EGL video sink
coreelements:  fakesink: Fake Sink
coreelements:  fdsink: Filedescriptor Sink
coreelements:  filesink: File Sink
xvimagesink:  xvimagesink: Video sink
-sh-4.4# 

[SGaist]

Which GStreamer version do you have ?

It's not mandatory but it might be easier to use with your custom pipeline.

[Jeffff]

Hi @SGaist , I found some log after I enable the qtmultimedia log by referring to: https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/How-to-use-qtmultimedia-QML-with-Gstreamer-1-0/ta-p/1118006
Here is the log that might be related:

qt.multimedia.video: source is VideoProducer(0x1930a08)
qt.multimedia.video: found videonode plugin "egl" 0x1941b90
qt.multimedia.video: found videonode plugin "imx6" 0x194cd20
qt.multimedia.video: media object is QObject(0x0)
qml: mainVideoProducer: onCompleted
Opening device: /dev/video1 
Cam Image Size: QSize(1280, 720)
qt.multimedia.video: Video surface format: QVideoSurfaceFormat(Format_RGB32, QSize(1280, 720), viewport=QRect(0,0 1280x720), pixelAspectRatio=QSize(1, 1), handleType=NoHandle, yCbCrColorSpace=YCbCr_Undefined)
    handleType = QVariant(QAbstractVideoBuffer::HandleType, )
    pixelFormat = QVariant(QVideoFrame::PixelFormat, )
    frameSize = QVariant(QSize, QSize(1280, 720))
    frameWidth = QVariant(int, 1280)
    viewport = QVariant(QRect, QRect(0,0 1280x720))
    scanLineDirection = QVariant(QVideoSurfaceFormat::Direction, )
    frameRate = QVariant(double, 0)
    pixelAspectRatio = QVariant(QSize, QSize(1, 1))
    sizeHint = QVariant(QSize, QSize(1280, 720))
    yCbCrColorSpace = QVariant(QVideoSurfaceFormat::YCbCrColorSpace, )
    mirrored = QVariant(bool, false) all supported formats: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21, Format_YUV420P, Format_YV12, Format_NV12, Format_NV21, Format_UYVY, Format_YUYV, Format_RGB32, Format_ARGB32, Format_BGR32, Format_BGRA32, Format_RGB565)
qt.multimedia.video: updatePaintNode: Video node created. Handle type: NoHandle  Supported formats for the handle by this node: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21)

On the other hand, if I run the "qmlvideo" which plays the video file. it is OK to play that video(avi format), and here is the log:

qt.multimedia.video: Video surface format: QVideoSurfaceFormat(Format_YUV420P, QSize(1920, 1080), viewport=QRect(0,0 1920x1080), pixelAspectRatio=QSize(1, 1), handleType=NoHandle, yCbCrColorSpace=YCbCr_Undefined)
    handleType = QVariant(QAbstractVideoBuffer::HandleType, )
    pixelFormat = QVariant(QVideoFrame::PixelFormat, )
    frameSize = QVariant(QSize, QSize(1920, 1080))
    frameWidth = QVariant(int, 1920)
    viewport = QVariant(QRect, QRect(0,0 1920x1080))
    scanLineDirection = QVariant(QVideoSurfaceFormat::Direction, )
    frameRate = QVariant(double, 30)
    pixelAspectRatio = QVariant(QSize, QSize(1, 1))
    sizeHint = QVariant(QSize, QSize(1920, 1080))
    yCbCrColorSpace = QVariant(QVideoSurfaceFormat::YCbCrColorSpace, )
    mirrored = QVariant(bool, false) all supported formats: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21, Format_YUV420P, Format_YV12, Format_NV12, Format_NV21, Format_UYVY, Format_YUYV, Format_RGB32, Format_ARGB32, Format_BGR32, Format_BGRA32, Format_RGB565)
qt.multimedia.video: updatePaintNode: Video node created. Handle type: NoHandle  Supported formats for the handle by this node: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21)

I notice that the frameRate is different, but in my side, I just feed the image one by one. And I have no idea how to set "frame rate". So any Idea what is wrong?

[SGaist]

How are you feeding your pipeline ?

[Jeffff]

Seems that I cannot upload my source code here. It is pretty simple, first I use opencv to capture a image, and emit the frame.

    forever{
        try {
            if (!normal_video.isOpened()) {
                continue;
            }
            cv::Mat normal_frame;
            normal_video >> normal_frame;
            emit updateCameraFrame(normal_frame.clone());
        } 

Then in another thread, the frame is handled:

void VideoManager::frameReceived(const cv::Mat &frame) {
    if(mRunning){
        cv::Mat image(frame);
        mCameraProducer->updateFrame(Mat2QImage(image));
    }
}
......
void VideoProducer::updateFrame(QImage image)
{
    if(image.size() != mFormat.frameSize())
    {
        qDebug() << "newSize: " << image.size();
        closeSurface();
        mFormat = QVideoSurfaceFormat( image.size(), QVideoFrame::PixelFormat::Format_RGB32);
        mSurface->start(mFormat);
    }
    mSurface->present( QVideoFrame( image ) );
}

As shown above, everything is working unless last step: mSurface->present(), which cannot really present the image/video on screen.

[SGaist]

@Jeffff said in video doesn't display on LCD screen:

VideoProducer

Is that one in a different thread than the main thread ?

[Jeffff]

Yes, sure. video capturing and displaying are 2 different threads, they are sync by signal.

[SGaist]

That I understood, my question is: does the displaying happen in the main thread of your application ? Not a secondary thread that you created.

[Jeffff]

Yes, it is the main thread of my application. As I've mentioned at the beginning, the same code can run on my Ubuntu.

[SGaist]

@Jeffff said in video doesn't display on LCD screen:

multimedia.video: media object is QObject(0x0)

This one looks pretty odd and is likely related to your 0 frame rate.

Can you test with a minimal QVideoWidget to compare ?

[Jeffff]

I did some test by qmlvideo sample App, which runs good on same imx6 board. As qmlvideo runs smoothly, it means my running environment is good.
On the other hand, I found that if I use small resolution image frame(such as 480*270), it is working. All frames are display on preset windows(800*480), obviously, it should be gstreamer which enlarges the frame. The very weird thing is that other large resolutions, such as (640*360, 1280*720, 1920*1080), are not OK to be displayed, in replace, the area keeps dark.
With the logging enabled , I can also find the gstreamer plugin inserted as below figure shows even though video cannot be displayed.

[Jeffff]

For my own app and the qmlvideo app, I also dump their lib dependency as below:

file:///mnt/hgfs/Share/Screenshot%20from%202021-12-01%2023-30-58.png
Another strange part to me: the Qt5Multimediea appears in my app, while it doesn't appear in qmlvideo example. As I described before, my app cannot display video frame, while qmlvideo can display...

[SGaist]

@Jeffff said in video doesn't display on LCD screen:

qmlvideo

Can you give a link to that example ?

[Jeffff]

@SGaist
Sorry I cannot get your idea.
The link here guides me to get the "qmlvideo" app. I havn't change any code of this app, it should be the original sample code of using QtMultimedia.