QVideoFrame from custom buffer instead of memcpy()

[malikcis]

Hi,
I created my own video sink inherited from QVideoSink in Qt 6. And I want to show content of this sink in QML side.
For this, I copy a custom buffer to a QVideoFrame like this:

void Producer::sendImageToQML(uchar *pData)
{    
    QVideoFrame video_frame(QVideoFrameFormat(QSize(3840, 2160),QVideoFrameFormat::Format_YUYV));
    if(!video_frame.map(QVideoFrame::WriteOnly))
    {
        qWarning() << "QVideoFrame is not writable";
        return;
    }    
    **memcpy(video_frame.bits(0), pData, 16588800);**
    video_frame.unmap();
    m_videoSink->setVideoFrame(video_frame);
}

This works fine but CPU usage is high because of the memcpy operation.
Is there a way to generate a QVideoFrame from a custom buffer pointer?
Thx,
PS: the rest of the code is prety much like this one.
Malikcis

[SGaist]

Hi,

You would have a copy operation in any case.

What is your data source ?

One thing to take into account, you are moving 4K images around, this has a cost.

[malikcis]

@SGaist Hi Samuel,
I really appreciate your feedback.

"What is your data source ?"
My data source is an embedded CSI camera that I access via native Gstreamer "appsink" API. This is needed because I need to dynamically configure Gstreamer. Therefore I cannot use QT Mediaplayer or FFMPEG.

"You would have a copy operation in any case."
Unfortunately I don't agree with you here because I actually already managed to demonstrate custom pointer to QVideoFrame conversion in QT5 using QAbstractVideoBuffer (Unfortunately, this is not available in QT6 anymore).
Here is the main code:

int Gstpipeline::sendNewSample2QML(uchar *pData)
{
    //gsize size;
    int video_with;
    int video_height;
    int video_size;
    int bytes_per_line;

    video_with = 480;
    video_height = 800;

    video_size = video_with * video_height * 4;
    bytes_per_line = video_with * 4;

    VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer.

    emit newFrameAvailable(QVideoFrame(vbuffer, QSize(video_with, video_height), QVideoFrame::Format_BGR32));

    return 0;
}

I subclassed QAbstractVideoBuffer as can be seen here:

//videoBuffer.h
#ifndef VIDEOBUFFER_H
#define VIDEOBUFFER_H

#include <QAbstractVideoBuffer>

class VideoBuffer : public QAbstractVideoBuffer
{
public:
    VideoBuffer(uchar *data, int numBytes, int bytesPerLine);

    virtual uchar *map(QAbstractVideoBuffer::MapMode mode, int *numBytes,
                       int *bytesPerLine) override;

    virtual QAbstractVideoBuffer::MapMode mapMode() const override;

    virtual void unmap() override;

    void setMap(uchar *data, int numBytes, int bytesPerLine) {
        m_data = data;
        m_numBytes = numBytes;
        m_bytesPerLine = bytesPerLine;
    }

private:
    uchar *m_data;
    int m_numBytes;
    int m_bytesPerLine;
};

#endif//VIDEOBUFFER_H

//############################
//videoBuffer.cpp
#include "videoBuffer.h"
#include <QDebug>

VideoBuffer::VideoBuffer(uchar *data, int numBytes, int bytesPerLine)
    : QAbstractVideoBuffer(QAbstractVideoBuffer::NoHandle)
    , m_data(data)
    , m_numBytes(numBytes)
    , m_bytesPerLine(bytesPerLine)
{

}

uchar *VideoBuffer::map(QAbstractVideoBuffer::MapMode mode, int *numBytes,
                        int *bytesPerLine)
{
    Q_UNUSED(mode)

    *numBytes = m_numBytes;
    *bytesPerLine = m_bytesPerLine;

    return m_data;
}

QAbstractVideoBuffer::MapMode VideoBuffer::mapMode() const
{
    return QAbstractVideoBuffer::ReadOnly;
}

void VideoBuffer::unmap()
{
   //delete m_data;
}

"One thing to take into account, you are moving 4K images around, this has a cost."
Note that only 480x800 pixel image is used here. When using memcpy on this small images, my CPU (NXP imx8mp) goes high up to 80% whereas with above code I only use 7%. This is the reason why I need alternative solution.

I was thinking of subclassing QVideoFrame in the same way as in the code above but I am not sure on how to do that. One will probably need to do a fake "mapping" that always uses the custom buffer.
Any hints or ideas will be welcome.
Thx, Malikcis

[Bonnie]

@malikcis According to qvideoframe.h, there is still a public constructor function using QAbstractVideoBuffer but commented as "internal" and also undocumented since QAbstractVideoBuffer is now in private headers.
I've answered in https://forum.qt.io/post/749829 about how to use QAbstractVideoBuffer in Qt6, maybe you can see if that helps.

[malikcis]

@Bonnie said in QVideoFrame from custom buffer instead of memcpy():

@malikcis According to qvideoframe.h, there is still a public constructor function using QAbstractVideoBuffer but commented as "internal" and also undocumented since QAbstractVideoBuffer is now in private headers.

Hi Bonnie,
Thank for this swift feedback. This is very exciting. I am going to read your post right now.

[malikcis]

@Bonnie Hi Bonnie,
I could implement your suggestion in https://forum.qt.io/post/749829 and I get a correct video. However my CPU usage is 100% even though QVideoFrame only reuses video pointer. I think QVideoFrame will do internal memcpy.
I put the entire project on github

As you can see in the code on github, I subclassed QAbstractVideoBuffer pretty much in the same way as in previously committed code and reused pData buffer:

 VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer.
    m_videoSink->setVideoFrame(QVideoFrame(vbuffer, QVideoFrameFormat(QSize(video_with, video_height),QVideoFrameFormat::Format_RGBX8888)));

Am I missing something?
THX

[Bonnie]

@malikcis Sorry I haven't digged into that so much, you could compare the source code of Qt5 and Qt6 to find the difference.

[malikcis]

@Bonnie Thank you,
I indeed compared the code and I see that some internal QT code changed in QT6.
Also it seems memory copy is done internally. Anyway, I will continue investigations.

[Riptide]

Hello @malikcis, do you found some solution to this topic? I have the same problem with qt6

[malikcis]

@Riptide
I am still using memcpy which consumes significant CPU resources.
I don't have alternative solution yet.

[malikcis]

@Riptide
Hi,
Did you manage to solve this issue?
I am still looking for better solutions.

One solution I would like to explore now has recently been suggested here:
https://community.nxp.com/t5/i-MX-Graphics-Knowledge-Base/QT6-Qml-embedded-video-play-for-weston-with-Gstreamer/ta-p/1825985