QVideoFrame problem with mapping memory from camera

[kolegs]

I got working example but only for iOS(gonna post it later, after cleaning my code) still dont know why Android doesnt work

[SGaist]

Unless I'm mistaken, on Android the data is an OpenGL texture that you might be able to access using handle which would return the texture id.

Hope it helps

[deion]

@SGaist

Some working code would be great ! I am not at all an expert on OpenGL and something that I could just copy paste (or start from a working and tested code on Android) would really really help. The camera is pretty useless on Android without being able to access the raw frame data :(

[GGenny]

Same problem... map fail on QVideoFrame from a QVideoFilterRunnable only in Android but... the image is displayed correctly.

Now i try the texture id workaround ( the handler is a GLTextureHandle )

[Dmitriy Bakhtiyarov]

I also have the same problem - on Windows my app work correctly, on Android QVideoFrame::map() always return false.

I was try to use QCamera + QAbstractVideoSurface, but on Android QCamera don't start - only change status to QCamera::LoadedStatus, not QCamera::ActiveStatus.

[Dmitriy Bakhtiyarov]

@kolegs I create ticket in qt bugtracker

and I recv message:
"It is normal. On different platforms the frames may come in different formats. In case of Android the frame contains an OpenGL texture handle, not raw pixel data, as GLTextureHandle indicates. To perform non-OpenGL operations on such an image on the CPU, the pixel data has to be read back first.
examples/multimedia/video/qmlvideofilter_opencl contains some code for this case too. Check rgbframehelper.h: here the imageWrapper function works both for plain (RGB) frames and ones that contain OpenGL textures. It should help getting started."

This example code really work, but very slow - I try to process every 10-th frame in video stream from camera, and I can see freezes in video playback.
I think that read-only processing may be asynchronous, but extracting image from opengl texture in another thread failed with segmentation fault(I think problem in opengl context). I continue research and write when fix this problem

[kolegs]

After all I decided to use QVideoProbe since I didn't need to change anything to the frame(I am using it to decode QR codes).

[GGenny]

GLTextureHandle workaround works nice as described in the example ( i copy directly the uchar without create QImage ). It should also work the patch ( I create a similar patch to avoid the workaround ) of qandroidvideorendercontrol.cpp, as described in the last patch of :

https://codereview.qt-project.org/#/c/124771/

[theshadowx]

@GGenny I have a problem with QVideoFrame and using OpenCV (I don't if I should create a thread for this?).

My camera takes images in YUV420P,.

QVideoFrame FilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, QVideoFilterRunnable::RunFlags flags)
{
    if(input->isValid()){
        input->map(QAbstractVideoBuffer::ReadWrite);
        /// QVideoFrame to cv::Mat
        Mat cvImg = Mat(input->height(),input->width(), CV_8UC3,input->bits(),input->bytesPerLine());

        /// Apply Filter
        Mat edges;

        /// to grayscale
        cvtColor(cvImg, edges, COLOR_YUV2GRAY_420);

        /// apply filters
        GaussianBlur(edges, edges, Size(7,7), 1.5, 1.5);
        Canny(edges, edges, 0, 30, 3);

        /// convert to YUV420
        cvtColor(edges,edges, COLOR_GRAY2RGB);
        cvtColor(edges,edges, COLOR_RGB2YUV_I420);

        /// .....   ???
    }

     return *input;
}

My question is how to return the modified frame ?

I tried to use QImage, but unfortunately it doesn't support YUV, so in return I have a blank frame as the surfaceFormat is YUV.

As @deion said, there should be more examples about these new features, otherwise they are "pretty USELESS"

If this question should be in a new thread, please let me know.

[Julian Guarin]

@theshadowx You must input->unmap() to write the buffer. Also check that map() returns true. Otherwise you would have to use OpenGl directly.

[rhardih]

Can anyone link to an example using an OpenGL texture instead of raw frame data?

I'm trying a similar approach as @theshadowx, where I'm passing the mapped bits() to OpenCV for processing. I'm not entirely clear how this would work in the case of the data being an OpenGL texture.

CC @SGaist @Julian-Guarin