How to get ROI of QVideoFrame?

[MHermann]

Hi all,

I have a QVideoFrame (640x480).
How can I extract a ROI of this frame?
Look at this example. How can I get a new QVideoFrame of ROI?

[JoeCFD]

call image() to get its image first and extract ROI from the image?
and QVideoFrame new_video_frame( ROI_image ).

[Pl45m4]

@MHermann

Convert QVideoFrame to QImage then access/grab the region of choice.
(mind the correct pixel format and buffer size)

[MHermann]

Thanks for the replies.
I tried the following:

• Convert frame to image => OK
• Get ROI-image => OK (I checked by saving and looking at the image)
• Convert ROI-image to ROI-frame => NOK (In GUI no image is shown, it is black. If I don't edit *input, than the image is shown correctly in GUI.)

I have to change the input frame so that the ROI-frame is shown in the GUI.
How can I do this?

QVideoFrame MyVideoFilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags)
{
   ...

   m_filter->m_frame.copyData(*input);

   QImage img((const uchar*)(m_filter->m_frame.data.data()), m_filter->m_frame.size.width(), m_filter->m_frame.size.height(), m_filter->m_frame.bytesPerLine, QImage::Format_RGB444);

   QImage imgRoi = img.copy(x, y, width, height);
   QVideoFrame frameRoi(imgRoi);

   *input = QVideoFrame();
   *input = frameRoi;
    
   return *input;

[jsulm]

@MHermann said in How to get ROI of QVideoFrame?:

Convert ROI-image to ROI-frame => NOK

In what way it is NOK?

"I have to change the input frame so that the ROI-frame is shown in the GUI." - what do you have to change exactly?

[MHermann]

@jsulm:
No image is shown in qml. It is black.
If I don't edit *input, than the image is shown correctly.

[jsulm]

@MHermann said in How to get ROI of QVideoFrame?:

*input = QVideoFrame();
*input = frameRoi;

1. The first line above is not needed (why do you create a new QVideoFrame, assign it to input and then assign frameRoi to input?). The input has to point to QVideoFrame instance already anyway if you want to assign another QVideoFrame to it.
2. Did you check whether the created QImage(s) are valid?

[MHermann]

@jsulm

Yes, I checked the imges. The images are correct.

[jsulm]

@MHermann What does https://doc.qt.io/qt-6/qvideoframe.html#isValid return?
It could also be that the QImage has to be converted to a supported video format first.

[MHermann]

@jsulm
The edited QVideoFrame is invalid ...

Any suggestions how I convert the QImage correctly to QVideoFrame?

[jsulm]

@MHermann You should check which formats QVideoFrame supports and what format you have in your QImage. Could be an issue converting different formats.
See the documentation (https://doc.qt.io/qt-6/qvideoframe.html#QVideoFrame-2):
"Otherwise, if the QImage::Format matches none of video formats, the image is first converted to a supported (A)RGB format using QImage::convertedTo() with the Qt::AutoColor flag."

[SGaist]

Why not just return frameRoi; ?

As @jsulm wrote, mutating the input frame does not make sense and is not how the video filter works AFAIK.

[MHermann]

@SGaist :
I tried that, but then it had no effect on the shown images in the GUI.
Thats why I think I have to do the changes on the input frame.

[SGaist]

Which version of Qt are you using ?

[MHermann]

@SGaist: I am using Qt 5.12.7

[SGaist]

There's an example application here that might help you get further.

[MHermann]

If somebody is interested in the soultion:

#include <QObject>
#include <QAbstractVideoFilter>
#include <QVideoFilterRunnable>
#include <QAbstractVideoFilter>
#include <QVideoFrame>
#include <QDebug>
#include <QThread>
#include <QFuture>
#include <QtConcurrent/QtConcurrent>
#include <QOpenGLContext>
#include <QOpenGLFunctions>
#include <QQmlContext>
#include "private/qvideoframe_p.h"

...

QImage MyVideoFilterRunnable::QVideoFrameToQImage(const QVideoFrame& videoFrame)
{
    if ( videoFrame.handleType() == QAbstractVideoBuffer::NoHandle )
    {
        QImage image = qt_imageFromVideoFrame( videoFrame );
        if ( image.isNull() )
        {
            return QImage();
        }

        if ( image.format() != QImage::Format_ARGB32 )
        {
            image = image.convertToFormat( QImage::Format_ARGB32 );
        }

        return image;
    }

    if ( videoFrame.handleType() == QAbstractVideoBuffer::GLTextureHandle )
    {
        QImage image( videoFrame.width(), videoFrame.height(), QImage::Format_ARGB32 );
        GLuint textureId = static_cast<GLuint>( videoFrame.handle().toInt() );
        QOpenGLContext* ctx = QOpenGLContext::currentContext();
        QOpenGLFunctions* f = ctx->functions();
        GLuint fbo;
        f->glGenFramebuffers( 1, &fbo );
        GLint prevFbo;
        f->glGetIntegerv( GL_FRAMEBUFFER_BINDING, &prevFbo );
        f->glBindFramebuffer( GL_FRAMEBUFFER, fbo );
        f->glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, textureId, 0 );
        f->glReadPixels( 0, 0,  videoFrame.width(),  videoFrame.height(), GL_RGBA, GL_UNSIGNED_BYTE, image.bits() );
        f->glBindFramebuffer( GL_FRAMEBUFFER, static_cast<GLuint>( prevFbo ) );
        return image.rgbSwapped();
    }

    return QImage();
}

QVideoFrame MyVideoFilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags)
{
   ...

   m_filter->m_frame.copyData(*input);

   QImage img = QVideoFrameToQImage( *input );

   QImage imgRoi = img.copy(xMin, yMin, width, height);
   QVideoFrame frameRoi = QVideoFrame(imgRoi);

   *input = frameRoi;

    m_filter->m_processThread = QtConcurrent::run(this, &MyVideoFilterRunnable::processVideoFrame, m_filter->m_frame);

    return *input;
}