QVideoFrame from custom buffer instead of memcpy()
-
wrote on 20 Jun 2023, 14:34 last edited by SGaist
Hi,
I created my own video sink inherited from QVideoSink in Qt 6. And I want to show content of this sink in QML side.
For this, I copy a custom buffer to a QVideoFrame like this:void Producer::sendImageToQML(uchar *pData) { QVideoFrame video_frame(QVideoFrameFormat(QSize(3840, 2160),QVideoFrameFormat::Format_YUYV)); if(!video_frame.map(QVideoFrame::WriteOnly)) { qWarning() << "QVideoFrame is not writable"; return; } **memcpy(video_frame.bits(0), pData, 16588800);** video_frame.unmap(); m_videoSink->setVideoFrame(video_frame); }
This works fine but CPU usage is high because of the memcpy operation.
Is there a way to generate a QVideoFrame from a custom buffer pointer?
Thx,
PS: the rest of the code is prety much like this one.
Malikcis -
Hi,
You would have a copy operation in any case.
What is your data source ?
One thing to take into account, you are moving 4K images around, this has a cost.
-
Hi,
You would have a copy operation in any case.
What is your data source ?
One thing to take into account, you are moving 4K images around, this has a cost.
wrote on 21 Jun 2023, 08:12 last edited by@SGaist Hi Samuel,
I really appreciate your feedback."What is your data source ?"
My data source is an embedded CSI camera that I access via native Gstreamer "appsink" API. This is needed because I need to dynamically configure Gstreamer. Therefore I cannot use QT Mediaplayer or FFMPEG."You would have a copy operation in any case."
Unfortunately I don't agree with you here because I actually already managed to demonstrate custom pointer to QVideoFrame conversion in QT5 using QAbstractVideoBuffer (Unfortunately, this is not available in QT6 anymore).
Here is the main code:int Gstpipeline::sendNewSample2QML(uchar *pData) { //gsize size; int video_with; int video_height; int video_size; int bytes_per_line; video_with = 480; video_height = 800; video_size = video_with * video_height * 4; bytes_per_line = video_with * 4; VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer. emit newFrameAvailable(QVideoFrame(vbuffer, QSize(video_with, video_height), QVideoFrame::Format_BGR32)); return 0; }
I subclassed QAbstractVideoBuffer as can be seen here:
//videoBuffer.h #ifndef VIDEOBUFFER_H #define VIDEOBUFFER_H #include <QAbstractVideoBuffer> class VideoBuffer : public QAbstractVideoBuffer { public: VideoBuffer(uchar *data, int numBytes, int bytesPerLine); virtual uchar *map(QAbstractVideoBuffer::MapMode mode, int *numBytes, int *bytesPerLine) override; virtual QAbstractVideoBuffer::MapMode mapMode() const override; virtual void unmap() override; void setMap(uchar *data, int numBytes, int bytesPerLine) { m_data = data; m_numBytes = numBytes; m_bytesPerLine = bytesPerLine; } private: uchar *m_data; int m_numBytes; int m_bytesPerLine; }; #endif//VIDEOBUFFER_H //############################ //videoBuffer.cpp #include "videoBuffer.h" #include <QDebug> VideoBuffer::VideoBuffer(uchar *data, int numBytes, int bytesPerLine) : QAbstractVideoBuffer(QAbstractVideoBuffer::NoHandle) , m_data(data) , m_numBytes(numBytes) , m_bytesPerLine(bytesPerLine) { } uchar *VideoBuffer::map(QAbstractVideoBuffer::MapMode mode, int *numBytes, int *bytesPerLine) { Q_UNUSED(mode) *numBytes = m_numBytes; *bytesPerLine = m_bytesPerLine; return m_data; } QAbstractVideoBuffer::MapMode VideoBuffer::mapMode() const { return QAbstractVideoBuffer::ReadOnly; } void VideoBuffer::unmap() { //delete m_data; }
"One thing to take into account, you are moving 4K images around, this has a cost."
Note that only 480x800 pixel image is used here. When using memcpy on this small images, my CPU (NXP imx8mp) goes high up to 80% whereas with above code I only use 7%. This is the reason why I need alternative solution.I was thinking of subclassing QVideoFrame in the same way as in the code above but I am not sure on how to do that. One will probably need to do a fake "mapping" that always uses the custom buffer.
Any hints or ideas will be welcome.
Thx, Malikcis -
@SGaist Hi Samuel,
I really appreciate your feedback."What is your data source ?"
My data source is an embedded CSI camera that I access via native Gstreamer "appsink" API. This is needed because I need to dynamically configure Gstreamer. Therefore I cannot use QT Mediaplayer or FFMPEG."You would have a copy operation in any case."
Unfortunately I don't agree with you here because I actually already managed to demonstrate custom pointer to QVideoFrame conversion in QT5 using QAbstractVideoBuffer (Unfortunately, this is not available in QT6 anymore).
Here is the main code:int Gstpipeline::sendNewSample2QML(uchar *pData) { //gsize size; int video_with; int video_height; int video_size; int bytes_per_line; video_with = 480; video_height = 800; video_size = video_with * video_height * 4; bytes_per_line = video_with * 4; VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer. emit newFrameAvailable(QVideoFrame(vbuffer, QSize(video_with, video_height), QVideoFrame::Format_BGR32)); return 0; }
I subclassed QAbstractVideoBuffer as can be seen here:
//videoBuffer.h #ifndef VIDEOBUFFER_H #define VIDEOBUFFER_H #include <QAbstractVideoBuffer> class VideoBuffer : public QAbstractVideoBuffer { public: VideoBuffer(uchar *data, int numBytes, int bytesPerLine); virtual uchar *map(QAbstractVideoBuffer::MapMode mode, int *numBytes, int *bytesPerLine) override; virtual QAbstractVideoBuffer::MapMode mapMode() const override; virtual void unmap() override; void setMap(uchar *data, int numBytes, int bytesPerLine) { m_data = data; m_numBytes = numBytes; m_bytesPerLine = bytesPerLine; } private: uchar *m_data; int m_numBytes; int m_bytesPerLine; }; #endif//VIDEOBUFFER_H //############################ //videoBuffer.cpp #include "videoBuffer.h" #include <QDebug> VideoBuffer::VideoBuffer(uchar *data, int numBytes, int bytesPerLine) : QAbstractVideoBuffer(QAbstractVideoBuffer::NoHandle) , m_data(data) , m_numBytes(numBytes) , m_bytesPerLine(bytesPerLine) { } uchar *VideoBuffer::map(QAbstractVideoBuffer::MapMode mode, int *numBytes, int *bytesPerLine) { Q_UNUSED(mode) *numBytes = m_numBytes; *bytesPerLine = m_bytesPerLine; return m_data; } QAbstractVideoBuffer::MapMode VideoBuffer::mapMode() const { return QAbstractVideoBuffer::ReadOnly; } void VideoBuffer::unmap() { //delete m_data; }
"One thing to take into account, you are moving 4K images around, this has a cost."
Note that only 480x800 pixel image is used here. When using memcpy on this small images, my CPU (NXP imx8mp) goes high up to 80% whereas with above code I only use 7%. This is the reason why I need alternative solution.I was thinking of subclassing QVideoFrame in the same way as in the code above but I am not sure on how to do that. One will probably need to do a fake "mapping" that always uses the custom buffer.
Any hints or ideas will be welcome.
Thx, Malikciswrote on 21 Jun 2023, 08:44 last edited by@malikcis According to
qvideoframe.h
, there is still a public constructor function usingQAbstractVideoBuffer
but commented as "internal" and also undocumented sinceQAbstractVideoBuffer
is now in private headers.
I've answered in https://forum.qt.io/post/749829 about how to useQAbstractVideoBuffer
in Qt6, maybe you can see if that helps. -
@malikcis According to
qvideoframe.h
, there is still a public constructor function usingQAbstractVideoBuffer
but commented as "internal" and also undocumented sinceQAbstractVideoBuffer
is now in private headers.
I've answered in https://forum.qt.io/post/749829 about how to useQAbstractVideoBuffer
in Qt6, maybe you can see if that helps.wrote on 21 Jun 2023, 09:21 last edited by@Bonnie said in QVideoFrame from custom buffer instead of memcpy():
@malikcis According to qvideoframe.h, there is still a public constructor function using QAbstractVideoBuffer but commented as "internal" and also undocumented since QAbstractVideoBuffer is now in private headers.
Hi Bonnie,
Thank for this swift feedback. This is very exciting. I am going to read your post right now. -
@malikcis According to
qvideoframe.h
, there is still a public constructor function usingQAbstractVideoBuffer
but commented as "internal" and also undocumented sinceQAbstractVideoBuffer
is now in private headers.
I've answered in https://forum.qt.io/post/749829 about how to useQAbstractVideoBuffer
in Qt6, maybe you can see if that helps.wrote on 22 Jun 2023, 16:10 last edited by malikcis@Bonnie Hi Bonnie,
I could implement your suggestion in https://forum.qt.io/post/749829 and I get a correct video. However my CPU usage is 100% even though QVideoFrame only reuses video pointer. I think QVideoFrame will do internal memcpy.
I put the entire project on githubAs you can see in the code on github, I subclassed QAbstractVideoBuffer pretty much in the same way as in previously committed code and reused pData buffer:
VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer. m_videoSink->setVideoFrame(QVideoFrame(vbuffer, QVideoFrameFormat(QSize(video_with, video_height),QVideoFrameFormat::Format_RGBX8888)));
Am I missing something?
THX -
@Bonnie Hi Bonnie,
I could implement your suggestion in https://forum.qt.io/post/749829 and I get a correct video. However my CPU usage is 100% even though QVideoFrame only reuses video pointer. I think QVideoFrame will do internal memcpy.
I put the entire project on githubAs you can see in the code on github, I subclassed QAbstractVideoBuffer pretty much in the same way as in previously committed code and reused pData buffer:
VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer. m_videoSink->setVideoFrame(QVideoFrame(vbuffer, QVideoFrameFormat(QSize(video_with, video_height),QVideoFrameFormat::Format_RGBX8888)));
Am I missing something?
THX -
@malikcis Sorry I haven't digged into that so much, you could compare the source code of Qt5 and Qt6 to find the difference.
-
@Bonnie Thank you,
I indeed compared the code and I see that some internal QT code changed in QT6.
Also it seems memory copy is done internally. Anyway, I will continue investigations. -
Hello @malikcis, do you found some solution to this topic? I have the same problem with qt6
-
Hello @malikcis, do you found some solution to this topic? I have the same problem with qt6
wrote on 9 Oct 2024, 14:20 last edited by@Riptide
Hi,
Did you manage to solve this issue?
I am still looking for better solutions.One solution I would like to explore now has recently been suggested here:
https://community.nxp.com/t5/i-MX-Graphics-Knowledge-Base/QT6-Qml-embedded-video-play-for-weston-with-Gstreamer/ta-p/1825985