Painting a QVideoFrame on a QVideoWidget with QT6

Spazz

Hey people,

I'm having trouble using the QVideoFrame class and especially its paint method.

I posted the same question on StackOverflow there, and I'm throwing a new bottle to the sea here :D ! The question is below.

Thanks for your help !

I'd like to process the stream of my webcam frame by frame with QT6. I've checked the internet but since QTMultimedia was heavily reworked with QT6, and since QT6 is pretty new, all the documentation/questions available are outdated.

So, In order to achieve my goal, I'm using a QMediaCaptureSession with a camera set on QMediaDevices::defaultVideoInput(). I checked that this was working by setting the video output of the QMediaCaptureSession to a QVideoWidget with m_session.setVideoOutput(ui->videowidget);, and it's working fine, except that I can't process the frames (basically, it's rendering my webcam on the QVideoWidget).

Now, to process the frames, I have to use a QVideoSink as far as I understand the documentation here and there. So I replaced m_session.setVideoOutput(ui->videowidget); with m_session.setVideoSink(&mysink);, where mysink is a QVideoSink.

Then, since I want to process the frames, I'm connecting the videoFrameChanged signal of mysink to a function processVideoFrame where I want to do 2 things :

process the current frame
render the result on the UI, ideally on ui->videowidget

This is the point where I'm struggling. I do not understand how to use the paint function of the class QVideoFrame to render the processed frame on the QVideoWidget. More precisely :

I do not understand how I am supposed to instantiate the QPainter. I tried a straightforward new QPainter(ui->videowidget) but it ends up in a QWidget::paintEngine: Should no longer be called exception and nothing is rendered
I do not understand what is actually representing the second parameter rect of QVideoFrame::paint?

I made a MWE, code is below.

mwe_videosinkpainting.h

#ifndef MWE_VIDEOSINKPAINTING_H
#define MWE_VIDEOSINKPAINTING_H

#include <QMainWindow>
#include <QMediaCaptureSession>
#include <QMediaDevices>
#include <QCamera>
#include <QVideoSink>
#include <QPainter>

QT_BEGIN_NAMESPACE
namespace Ui { class MWE_VideoSinkPainting; }
QT_END_NAMESPACE

class MWE_VideoSinkPainting : public QMainWindow
{
	Q_OBJECT

public:
	MWE_VideoSinkPainting(QWidget *parent = nullptr);
	~MWE_VideoSinkPainting();

private slots:
	void processVideoFrame();


private:
	Ui::MWE_VideoSinkPainting *ui;

	QVideoSink mysink;
	QMediaCaptureSession m_session;
	QScopedPointer<QCamera> m_camera;
};
#endif // MWE_VIDEOSINKPAINTING_H

mwe_videosinking.cpp

#include "mwe_videosinkpainting.h"
#include "ui_mwe_videosinkpainting.h"

MWE_VideoSinkPainting::MWE_VideoSinkPainting(QWidget *parent)
	: QMainWindow(parent)
	, ui(new Ui::MWE_VideoSinkPainting)
{
	ui->setupUi(this);

	m_camera.reset(new QCamera(QMediaDevices::defaultVideoInput()));
	m_session.setCamera(m_camera.data());
	//m_session.setVideoOutput(ui->videowidget);

	connect(&mysink, &QVideoSink::videoFrameChanged, this, &MWE_VideoSinkPainting::processVideoFrame);

	m_session.setVideoSink(&mysink);
	m_camera->start();
}

MWE_VideoSinkPainting::~MWE_VideoSinkPainting()
{
	delete ui;
}

void MWE_VideoSinkPainting::processVideoFrame()
{
	QVideoFrame videoframe = mysink.videoFrame();
	if(videoframe.map(QVideoFrame::ReadOnly))
	{
		//This is the part I'm struggling to understand and achieve
		videoframe.paint(new QPainter(ui->videowidget), QRectF(0.0f,0.0f,100.0f,100.0f), QVideoFrame::PaintOptions());
		videoframe.unmap();
	}
}

main.cpp

#include "mwe_videosinkpainting.h"

#include <QApplication>

int main(int argc, char *argv[])
{
	QApplication a(argc, argv);
	MWE_VideoSinkPainting w;
	w.show();
	return a.exec();
}

ui_mwe_videosinkpainting.h (just so that you have the whole code, it has no value for the question)

#ifndef UI_MWE_VIDEOSINKPAINTING_H
#define UI_MWE_VIDEOSINKPAINTING_H

#include <QtCore/QVariant>
#include <QtMultimediaWidgets/QVideoWidget>
#include <QtWidgets/QApplication>
#include <QtWidgets/QGridLayout>
#include <QtWidgets/QHBoxLayout>
#include <QtWidgets/QMainWindow>
#include <QtWidgets/QMenuBar>
#include <QtWidgets/QStatusBar>
#include <QtWidgets/QWidget>

QT_BEGIN_NAMESPACE

class Ui_MWE_VideoSinkPainting
{
public:
	QWidget *centralwidget;
	QGridLayout *gridLayout;
	QVideoWidget *videowidget;
	QHBoxLayout *horizontalLayout;
	QMenuBar *menubar;
	QStatusBar *statusbar;

	void setupUi(QMainWindow *MWE_VideoSinkPainting)
	{
		if (MWE_VideoSinkPainting->objectName().isEmpty())
			MWE_VideoSinkPainting->setObjectName(QString::fromUtf8("MWE_VideoSinkPainting"));
		MWE_VideoSinkPainting->resize(800, 600);
		centralwidget = new QWidget(MWE_VideoSinkPainting);
		centralwidget->setObjectName(QString::fromUtf8("centralwidget"));
		gridLayout = new QGridLayout(centralwidget);
		gridLayout->setObjectName(QString::fromUtf8("gridLayout"));
		videowidget = new QVideoWidget(centralwidget);
		videowidget->setObjectName(QString::fromUtf8("videowidget"));
		horizontalLayout = new QHBoxLayout(videowidget);
		horizontalLayout->setObjectName(QString::fromUtf8("horizontalLayout"));

		gridLayout->addWidget(videowidget, 0, 0, 1, 1);

		MWE_VideoSinkPainting->setCentralWidget(centralwidget);
		menubar = new QMenuBar(MWE_VideoSinkPainting);
		menubar->setObjectName(QString::fromUtf8("menubar"));
		menubar->setGeometry(QRect(0, 0, 800, 21));
		MWE_VideoSinkPainting->setMenuBar(menubar);
		statusbar = new QStatusBar(MWE_VideoSinkPainting);
		statusbar->setObjectName(QString::fromUtf8("statusbar"));
		MWE_VideoSinkPainting->setStatusBar(statusbar);

		retranslateUi(MWE_VideoSinkPainting);

		QMetaObject::connectSlotsByName(MWE_VideoSinkPainting);
	} // setupUi

	void retranslateUi(QMainWindow *MWE_VideoSinkPainting)
	{
		MWE_VideoSinkPainting->setWindowTitle(QCoreApplication::translate("MWE_VideoSinkPainting", "MWE_VideoSinkPainting", nullptr));
	} // retranslateUi

};

namespace Ui {
	class MWE_VideoSinkPainting: public Ui_MWE_VideoSinkPainting {};
} // namespace Ui

QT_END_NAMESPACE

#endif // UI_MWE_VIDEOSINKPAINTING_H

SGaist

Hi and welcome to devnet,

Just making an educated guess but:

QMediaCaptureSession has two things related to video:

setVideoSink
setVideoOutput

With the sink, you paint whatever you want on the frame and after that, it will be shown for you on the video widget.

You connect the videoFrameChanged signal on a slot, in the slot paint on it and you should be done.

Hope I am correct and it help.

Spazz

@Spazz said in Painting a QVideoFrame on a QVideoWidget with QT6:

mwe_videosinking.cpp

Thanks man, I needed this message. I don't know why I thought setVideoSink and setVideoOutput were exclusive.

For the people interested in the answer, basically the code I gave in origin post is the right one, you just need to uncomment the single comment in mwe_videosinking.cpp.

Also, note that you have to call setVideoSink BEFORE calling setVideoOutput.

Thanks again !

SGaist

@Spazz said in Painting a QVideoFrame on a QVideoWidget with QT6:

Also, note that you have to call setVideoSink BEFORE calling setVideoOutput.

That sounds like either a documentation or an implementation issue. I would recommend opening a ticket for it if none already exists.

for-lack-of-a-better-name-j

For those who are digging the depths of the internet to do this in Python, I figured out a way. It looks like the Python docs are wrong and/or ambiguous: "The video frame can then be used to read out the data of those frames and handle them further. When using QPainter, the QVideoFrame can be drawing using the paint() method in QVideoSink ."

I'm not a professional programmer so this may be ugly/non-conforming for some people but this works:

You can set the QMediaCaptureSession's video sink to your custom sink, and also make a QVideoWidget to display the frames. Then, take that custom sink, and connect some Slot method to the videoFrameChanged Signal, and inside that method do what you need to the frame, and then you can take that frame and use QVideoWidget.videoSink().setVideoFrame(the frame you just modified) and it works like a charm.

Please respond if this is unclear.

PS: mods/admins if this is in the wrong place please let me know.