Unsolved Problems with QMediaPlayer video frames
-
I'm using a QMediaPlayer to playback a video, and I need to process the video frames. Ideally I would like to process these frames on the GPU so I was hoping I could get access to the OpenGL textures associated with each frame.
I am having two problems:
First, the QVideoFrames generated by the player are not giving me access to the texture, and calling handleType() returns NoHandle.Second, when mapping the bits on a video frame, just to access the data, the format appears to be BGR24 instead of RGB24, even though my video surface only supports RGB24.
I wrote a class inheriting from QAbstractVideoSurface to act as a video output. I have a QMediaPlayer that is loading a video and have connected it to my video surface using setVideoOutput().
Here is the code:
/* TestVideoSurface.h */ #include <QAbstractVideoSurface> class TestVideoSurface : public QAbstractVideoSurface { Q_OBJECT public: bool present(const QVideoFrame &frame) override; QList<QVideoFrame::PixelFormat> supportedPixelFormats(QAbstractVideoBuffer::HandleType type = QAbstractVideoBuffer::NoHandle) const override; bool isFormatSupported(const QVideoSurfaceFormat &format) const override { return true; } };
/* TestVideoSurface.cpp */ #include "TestVideoSurface.h" bool TestVideoSurface::present(const QVideoFrame &frame) { QAbstractVideoBuffer::HandleType handleType = frame.handleType(); switch (handleType) { case QAbstractVideoBuffer::NoHandle: { // this case is always hit // CPU approach QVideoFrame frameCopy = frame; frameCopy.map(QAbstractVideoBuffer::ReadOnly); process_video_frame_cpu(frameCopy.bits()); frameCopy.unmap(); } break; case QAbstractVideoBuffer::GLTextureHandle: { // this case is never hit // GPU approach process_video_frame_gpu(frame.handle().toInt()); } break; default: break; } return true; } QList<QVideoFrame::PixelFormat> TestVideoSurface::supportedPixelFormats(QAbstractVideoBuffer::HandleType type) const { QList<QVideoFrame::PixelFormat> list; list << QVideoFrame::PixelFormat::Format_RGB24; return list; }
And to start playback:
qtPlayer.setVideoOutput(&myVideoSurface); qtPlayer.setMedia(QUrl::fromUserInput("data/test.mp4", QCoreApplication::applicationDirPath())); qtPlayer.play();
-
Hi,
The management of the format and type of the texture depends on the platform multimedia backend.
What OS are you running ? With which version of Qt ?