video doesn't display on LCD screen

Jeffff

Hi, I am using a NXP imx6 to do UVC camera capturing and screen displaying under QT. For now, I can get the image data(checked by printing part of video data out), however, the video cannot be displayed on screen as shown below.
As seen, the display area is always kept black. Here is the key code I use to display video:

//File: VideoProducer.cpp
void VideoProducer::updateFrame(QImage image)
{
    if(image.size() != mFormat.frameSize())
    {
        closeSurface();
        mFormat = QVideoSurfaceFormat( image.size(), QVideoFrame::PixelFormat::Format_RGB32);
        qDebug() << "Cam Image Size:" << mFormat.frameSize();
        if(!mSurface->start(mFormat)) {
            qDebug() << "VSurf Started Fail";
        }
    }
    
    QVideoFrame nframe(image);
    if(!nframe.isValid()){
        qDebug() << "QImage2QVideoFrame Err " ;
    }

    if(!mSurface->present( nframe )){
        qDebug() << "Present Fail";
    }
}

Meanwhile, in QML file side, it is written as below:

//VideoProducer.qml
import QtQuick 2.0
import QtMultimedia 5.9
import VideoProducer 1.0

Item {
    id: element
    property var mode: VideoOutput.PreserveAspectFit

    function getVideoProducer() {
        return videoProducer
    }
    function getVideoOutput() {
        return videoOutput
    }

    VideoProducer {
        id: videoProducer
    }

    VideoOutput {
        id: videoOutput
        anchors.fill: parent
        source: videoProducer
        fillMode: mode
    }
}

For the same code, it is running well on my PC(Ubuntu), I can display the camera content smoothly. However, it doesn't work on my embedded board, There is no error print in runtime log, I've also checked the libQt5Multimedia, it seems to be OK.
My QT Environment :QT5.10.1, I'm using Yocto for building. And I'm 100%sure the video input data has valid data. Any clue?

Jeffff

More information about my system. I'm using arm-gcc toolchain to compile the code for imx6 system. Everything seems OK, but for the last step:

mSurface->present( nframe )

it cannot display video even though this function returns true.

SGaist

Hi and welcome to devnet,

Can you get GStreamer to show the camera image directly ?
What are the specifications of the camera ?

Jeffff

@SGaist Very nice to get your reply:)
I can use gstreamer to show video on screen. However, due to UVC camera compatible reason, I cannot bridge camera with LCD directly with live display(by gst command). But I can output videos by using command(gst-launch-1.0 videotestsrc pattern=0 ! imxg2dvideosink). As far as I know, NXP has prepared some gstreamer plugin including camera, encoder and display, and my gstreamer are using these plugin(like imxg2dvideosink). I don't know whether my QT is adopting qst, and how QT works with gstreamer to display video, as well as what should I do to check or debug this part.
In my side, I've written my own code based on v4l2, which can output data(checked by converting data into JPG file), the image can also be converted to QImage format, which is the input of member function "VideoProducer::updateFrame" shown above

SGaist

Check with GStreamer's ~~qmlsink~~ qmlglsink, it might make your life easier.

Jeffff

@SGaist Can you introduce more detail about that? I'm quite a newbie in QT and don't get your point. Do you mean the QT Multimedia in imx6 must be using gstreamer for certain video processing(I guess such as scaling), and inside gstreamer, the plugin qmlsink if finally used for displaying? I searched the whole file system and I cannot get any file named with qmlsink. Also I searched my qtmultimedia source code by content, there is nothing found.
Meanwhile I find that there are 2 version of libgst existing in my target board, one is version 1.0, another is version0.1. I deleted all lib files of version 0.1, but nothing happens, the video still cannot be displayed on screen.

SGaist

Sorry, the name of the sink was wrong it's qmlglsink.

The QtMultimedia backend for Linux uses GStreamer, hower it has used it way before the qmlglsink existed and also it supports both the widgets and quick modules.

That said, it has no idea of any accelerated plugins like NXP is providing, hence using a custom pipeline is likely the best way to achieve what you want to do. The qmlglsink is part of the good plugin package of GStreamer however I don't remember in which version it has been made available so you should check that.

Jeffff

Thank you @SGaist .
Then I try to search my QtMultimedia by objdump:

It seems that this QtMultimedia doesn't use gstreamer. Also I searched my directory, no "qmlglsink" was found.
The QT version 5.10.1 is installed on this board. I really lost my direction.

SGaist

You are mixing things here.

QtMultimedia does not link to GStreamer. Its Linux plugin will be.
qmlglsink is part of GStreamer, but I don't remember since which version and it's part of one of the plugin packages of GStreamer.

Jeffff

I havn't found the "qmlglsink" plugin in my gstreamer1.0. Do you mean it is a must to use this plugin if we want to play video in QT?

-sh-4.4# gst-inspect-1.0 | grep sink
imxipu:  imxipuvideosink: Freescale IPU video sink
tcp:  tcpclientsink: TCP client sink
tcp:  tcpserversink: TCP server sink
tcp:  multifdsink: Multi filedescriptor sink
tcp:  multisocketsink: Multi socket sink
rtspclientsink:  rtspclientsink: RTSP RECORD client
imxv4l2video:  imxv4l2videosink: V4L2 CSI Video Sink
imxpxp:  imxpxpvideosink: Freescale PxP video sink
alsa:  alsasink: Audio sink (ALSA)
imxg2d:  imxg2dvideosink: Freescale G2D video sink
soup:  souphttpclientsink: HTTP client sink
udp:  udpsink: UDP packet sender
udp:  multiudpsink: UDP packet sender
udp:  dynudpsink: UDP packet sender
ximagesink:  ximagesink: Video sink
debug:  testsink: Test plugin
gio:  giosink: GIO sink
gio:  giostreamsink: GIO stream sink
playback:  playsink: Player Sink
autodetect:  autovideosink: Auto video sink
autodetect:  autoaudiosink: Auto audio sink
imxeglvivsink:  imxeglvivsink: Freescale EGL video sink
coreelements:  fakesink: Fake Sink
coreelements:  fdsink: Filedescriptor Sink
coreelements:  filesink: File Sink
xvimagesink:  xvimagesink: Video sink
-sh-4.4#

SGaist

Which GStreamer version do you have ?

It's not mandatory but it might be easier to use with your custom pipeline.

Jeffff

Hi @SGaist , I found some log after I enable the qtmultimedia log by referring to: https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/How-to-use-qtmultimedia-QML-with-Gstreamer-1-0/ta-p/1118006
Here is the log that might be related:

qt.multimedia.video: source is VideoProducer(0x1930a08)
qt.multimedia.video: found videonode plugin "egl" 0x1941b90
qt.multimedia.video: found videonode plugin "imx6" 0x194cd20
qt.multimedia.video: media object is QObject(0x0)
qml: mainVideoProducer: onCompleted
Opening device: /dev/video1 
Cam Image Size: QSize(1280, 720)
qt.multimedia.video: Video surface format: QVideoSurfaceFormat(Format_RGB32, QSize(1280, 720), viewport=QRect(0,0 1280x720), pixelAspectRatio=QSize(1, 1), handleType=NoHandle, yCbCrColorSpace=YCbCr_Undefined)
    handleType = QVariant(QAbstractVideoBuffer::HandleType, )
    pixelFormat = QVariant(QVideoFrame::PixelFormat, )
    frameSize = QVariant(QSize, QSize(1280, 720))
    frameWidth = QVariant(int, 1280)
    viewport = QVariant(QRect, QRect(0,0 1280x720))
    scanLineDirection = QVariant(QVideoSurfaceFormat::Direction, )
    frameRate = QVariant(double, 0)
    pixelAspectRatio = QVariant(QSize, QSize(1, 1))
    sizeHint = QVariant(QSize, QSize(1280, 720))
    yCbCrColorSpace = QVariant(QVideoSurfaceFormat::YCbCrColorSpace, )
    mirrored = QVariant(bool, false) all supported formats: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21, Format_YUV420P, Format_YV12, Format_NV12, Format_NV21, Format_UYVY, Format_YUYV, Format_RGB32, Format_ARGB32, Format_BGR32, Format_BGRA32, Format_RGB565)
qt.multimedia.video: updatePaintNode: Video node created. Handle type: NoHandle  Supported formats for the handle by this node: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21)

On the other hand, if I run the "qmlvideo" which plays the video file. it is OK to play that video(avi format), and here is the log:

qt.multimedia.video: Video surface format: QVideoSurfaceFormat(Format_YUV420P, QSize(1920, 1080), viewport=QRect(0,0 1920x1080), pixelAspectRatio=QSize(1, 1), handleType=NoHandle, yCbCrColorSpace=YCbCr_Undefined)
    handleType = QVariant(QAbstractVideoBuffer::HandleType, )
    pixelFormat = QVariant(QVideoFrame::PixelFormat, )
    frameSize = QVariant(QSize, QSize(1920, 1080))
    frameWidth = QVariant(int, 1920)
    viewport = QVariant(QRect, QRect(0,0 1920x1080))
    scanLineDirection = QVariant(QVideoSurfaceFormat::Direction, )
    frameRate = QVariant(double, 30)
    pixelAspectRatio = QVariant(QSize, QSize(1, 1))
    sizeHint = QVariant(QSize, QSize(1920, 1080))
    yCbCrColorSpace = QVariant(QVideoSurfaceFormat::YCbCrColorSpace, )
    mirrored = QVariant(bool, false) all supported formats: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21, Format_YUV420P, Format_YV12, Format_NV12, Format_NV21, Format_UYVY, Format_YUYV, Format_RGB32, Format_ARGB32, Format_BGR32, Format_BGRA32, Format_RGB565)
qt.multimedia.video: updatePaintNode: Video node created. Handle type: NoHandle  Supported formats for the handle by this node: (Format_ARGB32, Format_RGB32, Format_RGB565, Format_BGRA32, Format_BGR32, Format_YUV420P, Format_YV12, Format_UYVY, Format_YUYV, Format_NV12, Format_NV21)

I notice that the frameRate is different, but in my side, I just feed the image one by one. And I have no idea how to set "frame rate". So any Idea what is wrong?

SGaist

How are you feeding your pipeline ?

Jeffff

Seems that I cannot upload my source code here. It is pretty simple, first I use opencv to capture a image, and emit the frame.

    forever{
        try {
            if (!normal_video.isOpened()) {
                continue;
            }
            cv::Mat normal_frame;
            normal_video >> normal_frame;
            emit updateCameraFrame(normal_frame.clone());
        }

Then in another thread, the frame is handled:

void VideoManager::frameReceived(const cv::Mat &frame) {
    if(mRunning){
        cv::Mat image(frame);
        mCameraProducer->updateFrame(Mat2QImage(image));
    }
}
......
void VideoProducer::updateFrame(QImage image)
{
    if(image.size() != mFormat.frameSize())
    {
        qDebug() << "newSize: " << image.size();
        closeSurface();
        mFormat = QVideoSurfaceFormat( image.size(), QVideoFrame::PixelFormat::Format_RGB32);
        mSurface->start(mFormat);
    }
    mSurface->present( QVideoFrame( image ) );
}

As shown above, everything is working unless last step: mSurface->present(), which cannot really present the image/video on screen.

SGaist

@Jeffff said in video doesn't display on LCD screen:

VideoProducer

Is that one in a different thread than the main thread ?

Jeffff

Yes, sure. video capturing and displaying are 2 different threads, they are sync by signal.

SGaist

That I understood, my question is: does the displaying happen in the main thread of your application ? Not a secondary thread that you created.

Jeffff

Yes, it is the main thread of my application. As I've mentioned at the beginning, the same code can run on my Ubuntu.

SGaist

@Jeffff said in video doesn't display on LCD screen:

multimedia.video: media object is QObject(0x0)

This one looks pretty odd and is likely related to your 0 frame rate.

Can you test with a minimal QVideoWidget to compare ?

Jeffff

I did some test by qmlvideo sample App, which runs good on same imx6 board. As qmlvideo runs smoothly, it means my running environment is good.
On the other hand, I found that if I use small resolution image frame(such as 480*270), it is working. All frames are display on preset windows(800*480), obviously, it should be gstreamer which enlarges the frame. The very weird thing is that other large resolutions, such as (640*360, 1280*720, 1920*1080), are not OK to be displayed, in replace, the area keeps dark.
With the logging enabled , I can also find the gstreamer plugin inserted as below figure shows even though video cannot be displayed.