Converting between QVideoframe and Opencv Mat

Marc 2050

Hi.

I've been spending days reading forums and trying to figure out the formats and get my camera work on Android. Please help.

Input is QVideoframe and the format I checked is Format_BGR32.
So I created a mat with the codes - cv::Mat mbgra = cv::Mat(1080, 1920, CV_8UC4, input->bits());
Next, I drew a circle - cv::circle(mbgra, cv::Point(200, 200), 15, cv::Scalar(0,0,255), 2, CV_AA );
Now I try to convert back with the following codes:
@
cv::Mat *m = new cv::Mat;
cvtColor(mbgra, *m, cv::COLOR_BGR2RGBA);
QVideoFrame(new MyBuffer(m), QSize(mat.cols, mat.rows), QVideoFrame::Format_BGRA32); //
@
MyBuffer is a subclass of QAbstractVideoBuffer.

I simply could not get the image correctly.

Any pointer is much appreciated.

Thanks.
Marc

SGaist

Hi and welcome to devnet,

IIRC, QVideoFrame gets an OpenGL texture on Android. What is your exact goal with the images from the camera ?

Marc 2050

@SGaist
My goal is to apply some algorithms on the Mat data using OpenCV functions. The algorithms would chuck out some info which I would draw on the Mat data. After that, I would like this data to be sent back to the display. From user point of view, it is like seeing the camera with some lines and circles drawn on the camera video.

I had spent really a lot of time trying to make the format conversions. I am at my wits end.

J.Hilk

@Marc-2050 I'vae posted this before, but I still have the link.

Check this git project:
https://github.com/dbzhang800/QtOpenCV
very convenient class to make a QImage out of a mat and vice versa

However, if I see this correctly, it might not be exactly what you're looking for.

Marc 2050

@J.Hilk
Thanks. But that's not what I'm after. I had done Qimage to Mat and vice versa. I'm after the QVideoFrame.

I cannot understand how QT works for this QVideoframe. Specifically how do you take a Format_BGR32 and convert to Mat and then back to QVideoframe? I thought my codes should work but it didnt. I tried different permutation of CV_format and QVideoframe format. Nothing match. Is there anyone who tried successfully to convert Format_BGR32 back and forth from QVideoframe?

Thanks.
Marc

J.Hilk

@Marc-2050 you know, you can convert a QVideoframe into a QImage :) right?

Might not be the most direct way, but its one.

something like this:

QImage img( currentFrame.bits(),
             currentFrame.width(),
             currentFrame.height(),
             currentFrame.bytesPerLine(),
             QVideoFrame::imageFormatFromPixelFormat(currentFrame.pixelFormat()));

Marc 2050

@J.Hilk
Thanks! This is going through QImage instead of QVIdeoframe<->Mat directly...
But even when I try this method, I got strange behavior.
My codes as follow:
@
QImage img( input->bits(),
input->width(),
input->height(),
input->bytesPerLine(),
QVideoFrame::imageFormatFromPixelFormat(input->pixelFormat()));
cv::Mat mat(img.height(), img.width(), CV_8UC3, img.bits(),img.bytesPerLine());
cv::circle(mat, cv::Point(100, 100), 35, cv::Scalar(0,0,255), 2, CV_AA );
QImage tmp((uchar*)mat.data, mat.cols, mat.rows, QImage::Format_RGB888);
QVideoFrame *output = new QVideoFrame(tmp);
return *output;
@

The image displayed did not show the circle that I drew. What am I missing??

Thank you.
Marc

J.Hilk

@Marc-2050 Hard to tell, my experience with opencv is limited, and this is propbably the wrong forum for that. but did you make sure, that the circle is drawn on the mat correctly?
IIRC opencv has a save function. save the image and check if it's drawn or not.

PS:: for such simple operations as drawing shapes on an Image, Qt has its own reliable and fast functions.

QPainter p(image);
p.setRenderHints( QPainter.HighQualityAntialiasing )

p.setBrush( Qt.white );
p.setPen( QPen( Qt.green, 3.0 ) );
p.drawEllipse( QPoint( image.width()/2, image.height/2 ), image.width()/4, image.height/4 );
p.end();

Marc 2050

@J.Hilk
I tried the codes for QPainter.
It also turns out that there is nothing drawn on the image after the painting codes.

Does it have to do with the buffer? If the buffer is mapped as ReadOnly, would it cause no reporting error even though the drawing routines were executed?
I also tried ReadWrite as the buffer mode but it crashes on the Android.
So, I'm back to square one with a simple request of able to write some lines on the camera views on an Android device using QT.
If anyone out there solved this, I would deeply appreciate some help.

Thanks.
Marc

SGaist

Can you show your current full code for getting the video frame, drawing on it and then sending it forward ?

Did you also consider using QML ? If you only want to add overlay information on top of your video it might be simpler and more performant.

Marc 2050

@SGaist
Here's my codes -

QVideoFrame FaceRecognitionFilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags)
{
    input->map(QAbstractVideoBuffer::ReadOnly);
    QImage img( input->bits(),
             input->width(),
             input->height(),
             input->bytesPerLine(),
             QVideoFrame::imageFormatFromPixelFormat(input->pixelFormat()));
 cv::Mat mat(img.height(), img.width(), CV_8UC3, img.bits(),img.bytesPerLine());
    cv::circle(mat, cv::Point(100, 100), 35, cv::Scalar(0,0,255), 2, CV_AA );
	QImage tmp((uchar*)mat.data, mat.cols, mat.rows, QImage::Format_RGB888);
    QVideoFrame *output = new QVideoFrame(tmp);
    input->unmap(); 
return *output;
}

I realized that no matter what I did, even using the QPaint codes as suggested above, nothing would show on the output image. The codes would only work based on my original codes of declaring QVideframe via QAbstractVideoBuffer. Which then goes back to my question of how to deal with the formating of the image which is required in declaring the buffer format. How and what to do when a QVideframe is Format_BGR32 as input.

Marc 2050

@SGaist
Hi.
Could you elaborate on QML option and how to overlay information on top of the video? I'm using QML to display a videooutput from the camera.

SGaist

Take a look at the QML Video Example.