Problems with QMediaPlayer video frames



  • I'm using a QMediaPlayer to playback a video, and I need to process the video frames. Ideally I would like to process these frames on the GPU so I was hoping I could get access to the OpenGL textures associated with each frame.

    I am having two problems:
    First, the QVideoFrames generated by the player are not giving me access to the texture, and calling handleType() returns NoHandle.

    Second, when mapping the bits on a video frame, just to access the data, the format appears to be BGR24 instead of RGB24, even though my video surface only supports RGB24.

    I wrote a class inheriting from QAbstractVideoSurface to act as a video output. I have a QMediaPlayer that is loading a video and have connected it to my video surface using setVideoOutput().

    Here is the code:

    /* TestVideoSurface.h */
    #include <QAbstractVideoSurface>
    
    class TestVideoSurface : public QAbstractVideoSurface
    {
        Q_OBJECT
    
    public:
        bool present(const QVideoFrame &frame) override;
        QList<QVideoFrame::PixelFormat> supportedPixelFormats(QAbstractVideoBuffer::HandleType type = QAbstractVideoBuffer::NoHandle) const override;
        bool isFormatSupported(const QVideoSurfaceFormat &format) const override { return true; }
    };
    
    /* TestVideoSurface.cpp */
    
    #include "TestVideoSurface.h"
    
    bool TestVideoSurface::present(const QVideoFrame &frame)
    {
        QAbstractVideoBuffer::HandleType handleType = frame.handleType();
        switch (handleType)
        {
        case QAbstractVideoBuffer::NoHandle:
        {
            // this case is always hit
            // CPU approach
            QVideoFrame frameCopy = frame;
    
            frameCopy.map(QAbstractVideoBuffer::ReadOnly);
    
            process_video_frame_cpu(frameCopy.bits());
    
            frameCopy.unmap();
        }
    
            break;
        case QAbstractVideoBuffer::GLTextureHandle:
        {
            // this case is never hit
            // GPU approach
            process_video_frame_gpu(frame.handle().toInt());
        }
    
            break;
        default:
            break;
        }
    
        return true;
    }
    
    QList<QVideoFrame::PixelFormat> TestVideoSurface::supportedPixelFormats(QAbstractVideoBuffer::HandleType type) const
    {
        QList<QVideoFrame::PixelFormat> list;
        list << QVideoFrame::PixelFormat::Format_RGB24;
        return list;
    }
    

    And to start playback:

    qtPlayer.setVideoOutput(&myVideoSurface);
    qtPlayer.setMedia(QUrl::fromUserInput("data/test.mp4", QCoreApplication::applicationDirPath()));
    qtPlayer.play();
    

  • Lifetime Qt Champion

    Hi,

    The management of the format and type of the texture depends on the platform multimedia backend.

    What OS are you running ? With which version of Qt ?


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.