Unable to change video playback speed run-time QMediaplayer QAbstractVideoSurace

magicstar

I have implemented QAbstractVideoSurface as below and I am facing two issues:

Sometimes present method of QAbstractVideosurface is not called at all. There is no error as well. I can see video is playing from audio and timestamp getting updated.
By default video playback speed is 1.0. However, if I change it runtime, I see error in present function.
When media is stopped, I get access violation error for ui->m_GraphicsView.

Removing following condition solved 1st problem.

if ((surfaceFormat().pixelFormat() != frame.pixelFormat()) || (surfaceFormat().frameSize() != frame.size()))

video_frame_grabber.h

#include <QAbstractVideoSurface>
#include <QObject>
#include <QGraphicsView>
#include <QGraphicsScene>
#include <QPixmap>

class VideoFrameGrabber : public QAbstractVideoSurface
{
    Q_OBJECT
public:
    VideoFrameGrabber(QGraphicsView *view, QGraphicsPixmapItem *pixmap ,QObject *parent = nullptr);

    QList<QVideoFrame::PixelFormat> supportedPixelFormats(
            QAbstractVideoBuffer::HandleType handleType = QAbstractVideoBuffer::NoHandle) const;
    bool isFormatSupported(const QVideoSurfaceFormat &format) const;

    bool start(const QVideoSurfaceFormat &format);
    void stop();

    bool present(const QVideoFrame &frame);
    void displayframe(QImage img);
    QImage VideoFrameToImage(QVideoFrame frame);

private:
    QGraphicsView *view;
    QGraphicsPixmapItem *pixmap;
    QImage::Format imageFormat;
    int current_function;

signals:
    void frame_received();
};

video_frame_grabber.cpp

#include "video_frame_grabber.h"
#include <QtWidgets>
#include <qabstractvideosurface.h>
#include <qvideosurfaceformat.h>

VideoFrameGrabber::VideoFrameGrabber(QGraphicsView *view, QGraphicsPixmapItem *pixmap, QObject *parent)
    : QAbstractVideoSurface(parent)
    , view(view)
    , pixmap(pixmap)
    , imageFormat(QImage::Format_Invalid)
{
}

QList<QVideoFrame::PixelFormat> VideoFrameGrabber::supportedPixelFormats(QAbstractVideoBuffer::HandleType handleType) const
{
    Q_UNUSED(handleType);
    return QList<QVideoFrame::PixelFormat>()
        << QVideoFrame::Format_ARGB32
        << QVideoFrame::Format_ARGB32_Premultiplied
        << QVideoFrame::Format_RGB32
        << QVideoFrame::Format_RGB24
        << QVideoFrame::Format_RGB565
        << QVideoFrame::Format_RGB555
        << QVideoFrame::Format_ARGB8565_Premultiplied
        << QVideoFrame::Format_BGRA32
        << QVideoFrame::Format_BGRA32_Premultiplied
        << QVideoFrame::Format_BGR32
        << QVideoFrame::Format_BGR24
        << QVideoFrame::Format_BGR565
        << QVideoFrame::Format_BGR555
        << QVideoFrame::Format_BGRA5658_Premultiplied
        << QVideoFrame::Format_AYUV444
        << QVideoFrame::Format_AYUV444_Premultiplied
        << QVideoFrame::Format_YUV444
        << QVideoFrame::Format_YUV420P
        << QVideoFrame::Format_YV12
        << QVideoFrame::Format_UYVY
        << QVideoFrame::Format_YUYV
        << QVideoFrame::Format_NV12
        << QVideoFrame::Format_NV21
        << QVideoFrame::Format_IMC1
        << QVideoFrame::Format_IMC2
        << QVideoFrame::Format_IMC3
        << QVideoFrame::Format_IMC4
        << QVideoFrame::Format_Y8
        << QVideoFrame::Format_Y16
        << QVideoFrame::Format_Jpeg
        << QVideoFrame::Format_CameraRaw
        << QVideoFrame::Format_AdobeDng;
}

bool VideoFrameGrabber::isFormatSupported(const QVideoSurfaceFormat &format) const
{
    const QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(format.pixelFormat());
    const QSize size = format.frameSize();

    return imageFormat != QImage::Format_Invalid
            && !size.isEmpty()
            && format.handleType() == QAbstractVideoBuffer::NoHandle;
}

bool VideoFrameGrabber::start(const QVideoSurfaceFormat &format)
{
    const QImage::Format imageFormat = QVideoFrame::imageFormatFromPixelFormat(format.pixelFormat());
    const QSize size = format.frameSize();

    if (imageFormat != QImage::Format_Invalid && !size.isEmpty()) {
        this->imageFormat = imageFormat;
        QAbstractVideoSurface::start(format);
        return true;
    } 
    else{
        return false;
    }
}

void VideoFrameGrabber::stop()
{
    QAbstractVideoSurface::stop();

    view->update();
}

bool VideoFrameGrabber::present(const QVideoFrame &frame)
{
    if ((surfaceFormat().pixelFormat() != frame.pixelFormat()) || (surfaceFormat().frameSize() != frame.size())) {
        setError(IncorrectFormatError);
        stop();
        qDebug() << "error";
        return false;
    } 
    else {
        QImage img = VideoFrameToImage(frame);
        displayframe(img);
        emit frame_received();
        return true;
    }
}

void VideoFrameGrabber::displayframe(QImage img)
{
    pixmap->setPixmap(QPixmap::fromImage(img));
    view->fitInView(QRectF(0,0,img.width(),img.height()),Qt::KeepAspectRatio);
}

QImage VideoFrameGrabber::VideoFrameToImage(QVideoFrame frame)
{
    QVideoFrame copy(frame);
       if (copy.map(QAbstractVideoBuffer::ReadOnly)) {
           return QImage(frame.bits(), frame.width(), frame.height(), frame.bytesPerLine(), imageFormat);
       }
       return QImage();
}

mainwindow.cpp

m_MediaPlayer = new QMediaPlayer(this);
m_GraphicsScene = new QGraphicsScene();
pixmapItem = new QGraphicsPixmapItem();
m_GraphicsScene->addItem(pixmapItem);
ui->m_GraphicsView->setScene(m_GraphicsScene);
grabber = new VideoFrameGrabber(ui->m_GraphicsView, pixmapItem);
m_MediaPlayer->setVideoOutput(grabber);

m_MediaPlayer->setMedia(QUrl::fromLocalFile("1.mp4"));

ui->comboBox_videospeed->addItem("0.5x", QVariant(0.5));
ui->comboBox_videospeed->addItem("1.0x", QVariant(1.0));
ui->comboBox_videospeed->addItem("1.5x", QVariant(1.5));
ui->comboBox_videospeed->addItem("2.0x", QVariant(2.0));
ui->comboBox_videospeed->addItem("4.0x", QVariant(4.0));
ui->comboBox_videospeed->setCurrentIndex(1);

void MainWindow::on_comboBox_videospeed_currentIndexChanged(int index)
{
    qreal rate = ui->comboBox_videospeed->itemData(index).toDouble();
    m_MediaPlayer->setPlaybackRate(rate);
}

What am I missing in my implementation ?

SGaist

Hi,

That's likely because you call the stop method. You should rather handle the case where the size changes or the format changes.

Did you check which one it is that is triggering the stop call ?

magicstar

@SGaist ,
Yes. removing stop call works as expected. There is one more problem.
Sometimes, the present method is not called at all. Can you please suggest root cause for the issue?

SGaist

In what kind of circonstances ?

magicstar

@SGaist,
It is very random. I have cleaned code multiple times and rebuild. But, it still seems random. It does not seem issue of video format as well. Same video file may play sometimes and not play sometimes

SGaist

Can you see any particular changes in the video frame you receive when that happens ?

magicstar

@SGaist ,
The video frame is not displayed at all. As present function is not called. So, I do not see any change in videoframe.
I am not sure why I am facing many problems with above implementation. I have observed few other problems.

If I stop video or reload second video I get read access violation at
pixmap->setPixmap(QPixmap::fromImage(img));
I am unable to play video at 4x speed. audio and video is not synced at all. Seems like audio gets played at 4x speed and frames get piled up and are being displayed at its own speed.

I have implemented QAbstractVideoSurface to modify frames on the fly using different OpenCV functions. Currently, I am facing many problems just at the basic implementation. Is there any other better way to achieve the goal ?

SGaist

Did you check the stack trace to see what is happening ?
Are you sure your pixmap is value ?
Are creating a new VideoFrameGrabber in between ?
What kind of processing are you applying ? These can be time consuming and thus slow down the video pipeline.

magicstar

@SGaist ,

I tried debug problem through stack trace.
The read access violation is for QScrollBar::wheelEvent function of file Qt5Widgetsd.
I tried disabling QGraphicsView and pixmap event filter by setEnabled(false). But, it did not help.
I am not creating new VideoFrameGrabber in between. Also, there is no problem in QImage as i am able to save image.
Currently, I am not applying any processing. Just converting QVideoFrame To QImage

SGaist

Where do you have scrollbars in your application ?
What is the input format ?

magicstar

@SGaist
After lot of digging and modying code, I feel the problem is not with the scrollbars. After I stop the media, QGraphicsView *view seems to be deleted and that is why i get access violation error. However, I wonder if resources are deallocated when QAbstractVideoSurface::stop is called?
If so, what is the better way of implementing the functionality ?

SGaist

It should not...

Can you show the stack trace of the crash ?

magicstar

This post is deleted!

magicstar

@SGaist
Stack trace of the crash did help me with the error. Thanks a lot...
However, one error still remains. Present function of the QAbstractVideoSurface is not called sometimes. And this behaviour is very random as of now.

SGaist

What was it ?