Unable to change video playback speed run-time QMediaplayer QAbstractVideoSurace



  • I have implemented QAbstractVideoSurface as below and I am facing two issues:

    1. Sometimes present method of QAbstractVideosurface is not called at all. There is no error as well. I can see video is playing from audio and timestamp getting updated.

    2. By default video playback speed is 1.0. However, if I change it runtime, I see error in present function.

    If I remove following condition , everything works as expected

    if ((surfaceFormat().pixelFormat() != frame.pixelFormat()) || (surfaceFormat().frameSize() != frame.size()))
    

    video_frame_grabber.h

    #include <QAbstractVideoSurface>
    #include <QObject>
    #include <QGraphicsView>
    #include <QGraphicsScene>
    #include <QPixmap>
    
    class VideoFrameGrabber : public QAbstractVideoSurface
    {
        Q_OBJECT
    public:
        VideoFrameGrabber(QGraphicsView *view, QGraphicsPixmapItem *pixmap ,QObject *parent = nullptr);
    
        QList<QVideoFrame::PixelFormat> supportedPixelFormats(
                QAbstractVideoBuffer::HandleType handleType = QAbstractVideoBuffer::NoHandle) const;
        bool isFormatSupported(const QVideoSurfaceFormat &format) const;
    
        bool start(const QVideoSurfaceFormat &format);
        void stop();
    
        bool present(const QVideoFrame &frame);
        void displayframe(QImage img);
        QImage VideoFrameToImage(QVideoFrame frame);
    
    private:
        QGraphicsView *view;
        QGraphicsPixmapItem *pixmap;
        QImage::Format imageFormat;
        int current_function;
    
    signals:
        void frame_received();
    };
    

    video_frame_grabber.cpp

    #include "video_frame_grabber.h"
    #include <QtWidgets>
    #include <qabstractvideosurface.h>
    #include <qvideosurfaceformat.h>
    
    VideoFrameGrabber::VideoFrameGrabber(QGraphicsView *view, QGraphicsPixmapItem *pixmap, QObject *parent)
        : QAbstractVideoSurface(parent)
        , view(view)
        , pixmap(pixmap)
        , imageFormat(QImage::Format_Invalid)
    {
    }
    
    QList<QVideoFrame::PixelFormat> VideoFrameGrabber::supportedPixelFormats(QAbstractVideoBuffer::HandleType handleType) const
    {
        Q_UNUSED(handleType);
        return QList<QVideoFrame::PixelFormat>()
            << QVideoFrame::Format_ARGB32
            << QVideoFrame::Format_ARGB32_Premultiplied
            << QVideoFrame::Format_RGB32
            << QVideoFrame::Format_RGB24
            << QVideoFrame::Format_RGB565
            << QVideoFrame::Format_RGB555
            << QVideoFrame::Format_ARGB8565_Premultiplied
            << QVideoFrame::Format_BGRA32
            << QVideoFrame::Format_BGRA32_Premultiplied
            << QVideoFrame::Format_BGR32
            << QVideoFrame::Format_BGR24
            << QVideoFrame::Format_BGR565
            << QVideoFrame::Format_BGR555
            << QVideoFrame::Format_BGRA5658_Premultiplied
            << QVideoFrame::Format_AYUV444
            << QVideoFrame::Format_AYUV444_Premultiplied
            << QVideoFrame::Format_YUV444
            << QVideoFrame::Format_YUV420P
            << QVideoFrame::Format_YV12
            << QVideoFrame::Format_UYVY
            << QVideoFrame::Format_YUYV
            << QVideoFrame::Format_NV12
            << QVideoFrame::Format_NV21
            << QVideoFrame::Format_IMC1
            << QVideoFrame::Format_IMC2
            << QVideoFrame::Format_IMC3
            << QVideoFrame::Format_IMC4
            << QVideoFrame::Format_Y8
            << QVideoFrame::Format_Y16
            << QVideoFrame::Format_Jpeg
            << QVideoFrame::Format_CameraRaw
            << QVideoFrame::Format_AdobeDng;
    }
    
    bool VideoFrameGrabber::isFormatSupported(const QVideoSurfaceFormat &format) const
    {
        const QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(format.pixelFormat());
        const QSize size = format.frameSize();
    
        return imageFormat != QImage::Format_Invalid
                && !size.isEmpty()
                && format.handleType() == QAbstractVideoBuffer::NoHandle;
    }
    
    bool VideoFrameGrabber::start(const QVideoSurfaceFormat &format)
    {
        const QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(format.pixelFormat());
        const QSize size = format.frameSize();
    
        if (imageFormat != QImage::Format_Invalid && !size.isEmpty()) {
            this->imageFormat = imageFormat;
            QAbstractVideoSurface::start(format);
            return true;
        } 
        else{
            return false;
        }
    }
    
    void VideoFrameGrabber::stop()
    {
        QAbstractVideoSurface::stop();
    
        view->update();
    }
    
    bool VideoFrameGrabber::present(const QVideoFrame &frame)
    {
        if ((surfaceFormat().pixelFormat() != frame.pixelFormat()) || (surfaceFormat().frameSize() != frame.size())) {
            setError(IncorrectFormatError);
            stop();
            qDebug() << "error";
            return false;
        } 
        else {
            QImage img = VideoFrameToImage(frame);
            displayframe(img);
            emit frame_received();
            return true;
        }
    }
    
    void VideoFrameGrabber::displayframe(QImage img)
    {
        pixmap->setPixmap(QPixmap::fromImage(img));
        view->fitInView(QRectF(0,0,img.width(),img.height()),Qt::KeepAspectRatio);
    }
    
    QImage VideoFrameGrabber::VideoFrameToImage(QVideoFrame frame)
    {
        QVideoFrame copy(frame);
           if (copy.map(QAbstractVideoBuffer::ReadOnly)) {
               return QImage(frame.bits(), frame.width(), frame.height(), frame.bytesPerLine(), imageFormat);
           }
           return QImage();
    }
    

    mainwindow.cpp

    m_MediaPlayer = new QMediaPlayer(this);
    m_GraphicsScene = new QGraphicsScene();
    pixmapItem = new QGraphicsPixmapItem();
    m_GraphicsScene->addItem(pixmapItem);
    ui->m_GraphicsView->setScene(m_GraphicsScene);
    grabber = new VideoFrameGrabber(ui->m_GraphicsView, pixmapItem);
    m_MediaPlayer->setVideoOutput(grabber);
    
    m_MediaPlayer->setMedia(QUrl::fromLocalFile("1.mp4"));
    
    ui->comboBox_videospeed->addItem("0.5x", QVariant(0.5));
    ui->comboBox_videospeed->addItem("1.0x", QVariant(1.0));
    ui->comboBox_videospeed->addItem("1.5x", QVariant(1.5));
    ui->comboBox_videospeed->addItem("2.0x", QVariant(2.0));
    ui->comboBox_videospeed->addItem("4.0x", QVariant(4.0));
    ui->comboBox_videospeed->setCurrentIndex(1);
    
    void MainWindow::on_comboBox_videospeed_currentIndexChanged(int index)
    {
        qreal rate = ui->comboBox_videospeed->itemData(index).toDouble();
        m_MediaPlayer->setPlaybackRate(rate);
    }
    

    What am I missing in my implementation ?


  • Lifetime Qt Champion

    Hi,

    That's likely because you call the stop method. You should rather handle the case where the size changes or the format changes.

    Did you check which one it is that is triggering the stop call ?



  • @SGaist ,
    Yes. removing stop call works as expected. There is one more problem.
    Sometimes, the present method is not called at all. Can you please suggest root cause for the issue?


  • Lifetime Qt Champion

    In what kind of circonstances ?



  • @SGaist,
    It is very random. I have cleaned code multiple times and rebuild. But, it still seems random. It does not seem issue of video format as well. Same video file may play sometimes and not play sometimes


  • Lifetime Qt Champion

    Can you see any particular changes in the video frame you receive when that happens ?



  • @SGaist ,
    The video frame is not displayed at all. As present function is not called. So, I do not see any change in videoframe.
    I am not sure why I am facing many problems with above implementation. I have observed few other problems.

    1. If I stop video or reload second video I get read access violation at
      pixmap->setPixmap(QPixmap::fromImage(img));

    2. I am unable to play video at 4x speed. audio and video is not synced at all. Seems like audio gets played at 4x speed and frames get piled up and are being displayed at its own speed.

    I have implemented QAbstractVideoSurface to modify frames on the fly using different OpenCV functions. Currently, I am facing many problems just at the basic implementation. Is there any other better way to achieve the goal ?


  • Lifetime Qt Champion

    1. Did you check the stack trace to see what is happening ?
      Are you sure your pixmap is value ?
      Are creating a new VideoFrameGrabber in between ?
    2. What kind of processing are you applying ? These can be time consuming and thus slow down the video pipeline.

Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.