video doesn't display on LCD screen

SGaist

You are mixing things here.

QtMultimedia does not link to GStreamer. Its Linux plugin will be.
qmlglsink is part of GStreamer, but I don't remember since which version and it's part of one of the plugin packages of GStreamer.

Jeffff

I havn't found the "qmlglsink" plugin in my gstreamer1.0. Do you mean it is a must to use this plugin if we want to play video in QT?

-sh-4.4# gst-inspect-1.0 | grep sink
imxipu:  imxipuvideosink: Freescale IPU video sink
tcp:  tcpclientsink: TCP client sink
tcp:  tcpserversink: TCP server sink
tcp:  multifdsink: Multi filedescriptor sink
tcp:  multisocketsink: Multi socket sink
rtspclientsink:  rtspclientsink: RTSP RECORD client
imxv4l2video:  imxv4l2videosink: V4L2 CSI Video Sink
imxpxp:  imxpxpvideosink: Freescale PxP video sink
alsa:  alsasink: Audio sink (ALSA)
imxg2d:  imxg2dvideosink: Freescale G2D video sink
soup:  souphttpclientsink: HTTP client sink
udp:  udpsink: UDP packet sender
udp:  multiudpsink: UDP packet sender
udp:  dynudpsink: UDP packet sender
ximagesink:  ximagesink: Video sink
debug:  testsink: Test plugin
gio:  giosink: GIO sink
gio:  giostreamsink: GIO stream sink
playback:  playsink: Player Sink
autodetect:  autovideosink: Auto video sink
autodetect:  autoaudiosink: Auto audio sink
imxeglvivsink:  imxeglvivsink: Freescale EGL video sink
coreelements:  fakesink: Fake Sink
coreelements:  fdsink: Filedescriptor Sink
coreelements:  filesink: File Sink
xvimagesink:  xvimagesink: Video sink
-sh-4.4#

SGaist

Which GStreamer version do you have ?

It's not mandatory but it might be easier to use with your custom pipeline.

Jeffff

Hi @SGaist , I found some log after I enable the qtmultimedia log by referring to: https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/How-to-use-qtmultimedia-QML-with-Gstreamer-1-0/ta-p/1118006
Here is the log that might be related:

qt.multimedia.video: source is VideoProducer(0x1930a08)
qt.multimedia.video: found videonode plugin "egl" 0x1941b90
qt.multimedia.video: found videonode plugin "imx6" 0x194cd20
qt.multimedia.video: media object is QObject(0x0)
qml: mainVideoProducer: onCompleted
Opening device: /dev/video1 
Cam Image Size: QSize(1280, 720)
qt.multimedia.video: Video surface format: QVideoSurfaceFormat(Format_RGB32, QSize(1280, 720), viewport=QRect(0,0 1280x720), pixelAspectRatio=QSize(1, 1), handleType=NoHandle, yCbCrColorSpace=YCbCr_Undefined)
    handleType = QVariant(QAbstractVideoBuffer::HandleType, )
    pixelFormat = QVariant(QVideoFrame::PixelFormat, )
    frameSize = QVariant(QSize, QSize(1280, 720))
    frameWidth = QVariant(int, 1280)
    viewport = QVariant(QRect, QRect(0,0 1280x720))
    scanLineDirection = QVariant(QVideoSurfaceFormat::Direction, )
    frameRate = QVariant(double, 0)
    pixelAspectRatio = QVariant(QSize, QSize(1, 1))
    sizeHint = QVariant(QSize, QSize(1280, 720))
    yCbCrColorSpace = QVariant(QVideoSurfaceFormat::YCbCrColorSpace, )
    mirrored = QVariant(bool, false) all supported formats: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21, Format_YUV420P, Format_YV12, Format_NV12, Format_NV21, Format_UYVY, Format_YUYV, Format_RGB32, Format_ARGB32, Format_BGR32, Format_BGRA32, Format_RGB565)
qt.multimedia.video: updatePaintNode: Video node created. Handle type: NoHandle  Supported formats for the handle by this node: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21)

On the other hand, if I run the "qmlvideo" which plays the video file. it is OK to play that video(avi format), and here is the log:

qt.multimedia.video: Video surface format: QVideoSurfaceFormat(Format_YUV420P, QSize(1920, 1080), viewport=QRect(0,0 1920x1080), pixelAspectRatio=QSize(1, 1), handleType=NoHandle, yCbCrColorSpace=YCbCr_Undefined)
    handleType = QVariant(QAbstractVideoBuffer::HandleType, )
    pixelFormat = QVariant(QVideoFrame::PixelFormat, )
    frameSize = QVariant(QSize, QSize(1920, 1080))
    frameWidth = QVariant(int, 1920)
    viewport = QVariant(QRect, QRect(0,0 1920x1080))
    scanLineDirection = QVariant(QVideoSurfaceFormat::Direction, )
    frameRate = QVariant(double, 30)
    pixelAspectRatio = QVariant(QSize, QSize(1, 1))
    sizeHint = QVariant(QSize, QSize(1920, 1080))
    yCbCrColorSpace = QVariant(QVideoSurfaceFormat::YCbCrColorSpace, )
    mirrored = QVariant(bool, false) all supported formats: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21, Format_YUV420P, Format_YV12, Format_NV12, Format_NV21, Format_UYVY, Format_YUYV, Format_RGB32, Format_ARGB32, Format_BGR32, Format_BGRA32, Format_RGB565)
qt.multimedia.video: updatePaintNode: Video node created. Handle type: NoHandle  Supported formats for the handle by this node: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21)

I notice that the frameRate is different, but in my side, I just feed the image one by one. And I have no idea how to set "frame rate". So any Idea what is wrong?

SGaist

How are you feeding your pipeline ?

Jeffff

Seems that I cannot upload my source code here. It is pretty simple, first I use opencv to capture a image, and emit the frame.

    forever{
        try {
            if (!normal_video.isOpened()) {
                continue;
            }
            cv::Mat normal_frame;
            normal_video >> normal_frame;
            emit updateCameraFrame(normal_frame.clone());
        }

Then in another thread, the frame is handled:

void VideoManager::frameReceived(const cv::Mat &frame) {
    if(mRunning){
        cv::Mat image(frame);
        mCameraProducer->updateFrame(Mat2QImage(image));
    }
}
......
void VideoProducer::updateFrame(QImage image)
{
    if(image.size() != mFormat.frameSize())
    {
        qDebug() << "newSize: " << image.size();
        closeSurface();
        mFormat = QVideoSurfaceFormat( image.size(), QVideoFrame::PixelFormat::Format_RGB32);
        mSurface->start(mFormat);
    }
    mSurface->present( QVideoFrame( image ) );
}

As shown above, everything is working unless last step: mSurface->present(), which cannot really present the image/video on screen.

SGaist

@Jeffff said in video doesn't display on LCD screen:

VideoProducer

Is that one in a different thread than the main thread ?

Jeffff

Yes, sure. video capturing and displaying are 2 different threads, they are sync by signal.

SGaist

That I understood, my question is: does the displaying happen in the main thread of your application ? Not a secondary thread that you created.

Jeffff

Yes, it is the main thread of my application. As I've mentioned at the beginning, the same code can run on my Ubuntu.

SGaist

@Jeffff said in video doesn't display on LCD screen:

multimedia.video: media object is QObject(0x0)

This one looks pretty odd and is likely related to your 0 frame rate.

Can you test with a minimal QVideoWidget to compare ?

Jeffff

I did some test by qmlvideo sample App, which runs good on same imx6 board. As qmlvideo runs smoothly, it means my running environment is good.
On the other hand, I found that if I use small resolution image frame(such as 480*270), it is working. All frames are display on preset windows(800*480), obviously, it should be gstreamer which enlarges the frame. The very weird thing is that other large resolutions, such as (640*360, 1280*720, 1920*1080), are not OK to be displayed, in replace, the area keeps dark.
With the logging enabled , I can also find the gstreamer plugin inserted as below figure shows even though video cannot be displayed.

Jeffff

For my own app and the qmlvideo app, I also dump their lib dependency as below:

file:///mnt/hgfs/Share/Screenshot%20from%202021-12-01%2023-30-58.png
Another strange part to me: the Qt5Multimediea appears in my app, while it doesn't appear in qmlvideo example. As I described before, my app cannot display video frame, while qmlvideo can display...

SGaist

@Jeffff said in video doesn't display on LCD screen:

qmlvideo

Can you give a link to that example ?

Jeffff

@SGaist
Sorry I cannot get your idea.
The link here guides me to get the "qmlvideo" app. I havn't change any code of this app, it should be the original sample code of using QtMultimedia.

SGaist

Ok, the sample does not link to the QtMultimedia library directly because it does not use it directly as is your C++ application.

The QVideoWidget class, if memory serves well does not support the same number of formats as the QtQuick Video item.