QVideoFrame problem with mapping memory from camera

[deion]

I am having the exact same problem: I am trying to get raw (YUV or RGB) pixel data from the camera in Android. I tried it before in previous versions of Android and it did not work. Now Qt 5.5 is supposed to bring various improvements to the camera, however it does not work. I've used code similar to the one kolegs posted and I get the same results; the only diff is that I check input->handleType() and on Winodws PC this is QAbstractVideoBuffer::NoHandle and the map() function succeeds, while on Android (Galaxy Nexus phone running Android 4.3) handleType() is QAbstractVideoBuffer::GLTextureHandle and the map() function fails...

I really really wished Digia would release some working example on Android on how to simply capture frame data from the camera. The working qml camera example is pretty USELESS !!!!!

[kolegs]

I got working example but only for iOS(gonna post it later, after cleaning my code) still dont know why Android doesnt work

[SGaist]

Unless I'm mistaken, on Android the data is an OpenGL texture that you might be able to access using handle which would return the texture id.

Hope it helps

[deion]

@SGaist

Some working code would be great ! I am not at all an expert on OpenGL and something that I could just copy paste (or start from a working and tested code on Android) would really really help. The camera is pretty useless on Android without being able to access the raw frame data :(

[GGenny]

Same problem... map fail on QVideoFrame from a QVideoFilterRunnable only in Android but... the image is displayed correctly.

Now i try the texture id workaround ( the handler is a GLTextureHandle )

[Dmitriy Bakhtiyarov]

I also have the same problem - on Windows my app work correctly, on Android QVideoFrame::map() always return false.

I was try to use QCamera + QAbstractVideoSurface, but on Android QCamera don't start - only change status to QCamera::LoadedStatus, not QCamera::ActiveStatus.

[Dmitriy Bakhtiyarov]

@kolegs I create ticket in qt bugtracker

and I recv message:
"It is normal. On different platforms the frames may come in different formats. In case of Android the frame contains an OpenGL texture handle, not raw pixel data, as GLTextureHandle indicates. To perform non-OpenGL operations on such an image on the CPU, the pixel data has to be read back first.
examples/multimedia/video/qmlvideofilter_opencl contains some code for this case too. Check rgbframehelper.h: here the imageWrapper function works both for plain (RGB) frames and ones that contain OpenGL textures. It should help getting started."

This example code really work, but very slow - I try to process every 10-th frame in video stream from camera, and I can see freezes in video playback.
I think that read-only processing may be asynchronous, but extracting image from opengl texture in another thread failed with segmentation fault(I think problem in opengl context). I continue research and write when fix this problem

[kolegs]

After all I decided to use QVideoProbe since I didn't need to change anything to the frame(I am using it to decode QR codes).

[GGenny]

GLTextureHandle workaround works nice as described in the example ( i copy directly the uchar without create QImage ). It should also work the patch ( I create a similar patch to avoid the workaround ) of qandroidvideorendercontrol.cpp, as described in the last patch of :

https://codereview.qt-project.org/#/c/124771/

[theshadowx]

@GGenny I have a problem with QVideoFrame and using OpenCV (I don't if I should create a thread for this?).

My camera takes images in YUV420P,.

QVideoFrame FilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, QVideoFilterRunnable::RunFlags flags)
{
    if(input->isValid()){
        input->map(QAbstractVideoBuffer::ReadWrite);
        /// QVideoFrame to cv::Mat
        Mat cvImg = Mat(input->height(),input->width(), CV_8UC3,input->bits(),input->bytesPerLine());

        /// Apply Filter
        Mat edges;

        /// to grayscale
        cvtColor(cvImg, edges, COLOR_YUV2GRAY_420);

        /// apply filters
        GaussianBlur(edges, edges, Size(7,7), 1.5, 1.5);
        Canny(edges, edges, 0, 30, 3);

        /// convert to YUV420
        cvtColor(edges,edges, COLOR_GRAY2RGB);
        cvtColor(edges,edges, COLOR_RGB2YUV_I420);

        /// .....   ???
    }

     return *input;
}

My question is how to return the modified frame ?

I tried to use QImage, but unfortunately it doesn't support YUV, so in return I have a blank frame as the surfaceFormat is YUV.

As @deion said, there should be more examples about these new features, otherwise they are "pretty USELESS"

If this question should be in a new thread, please let me know.

[Julian Guarin]

@theshadowx You must input->unmap() to write the buffer. Also check that map() returns true. Otherwise you would have to use OpenGl directly.

[rhardih]

Can anyone link to an example using an OpenGL texture instead of raw frame data?

I'm trying a similar approach as @theshadowx, where I'm passing the mapped bits() to OpenCV for processing. I'm not entirely clear how this would work in the case of the data being an OpenGL texture.

CC @SGaist @Julian-Guarin