QImage manipulation crashes on android

[SGaist]

Hi,

Did you check what type of data you got with your QVideoFrame ? It might be an OpenGL texture, is that a supported case by your videodecoder object ?

[Julian Guarin]

@SGaist Thank you.

I will check, nevertheless I'm checking the pixel format... which is BGR32 in the android run it returns (QVideoFrame::Format_BGR32), on the Mac it returns (QVideoFrame::Format_NV12).

So the decoding function is for Android case is:

void qt_convert_BGR32_to_ARGB32(const uchar *pinframe, uchar *poutframe, int width, int height, int flags){
    Q_UNUSED(flags);
    for (int i = 0; i < height; i++){
        for (int j = 0; j < width; j ++, pinframe +=4, poutframe +=4){
            poutframe[0] = pinframe[3];
            poutframe[1] = pinframe[2];
            poutframe[2] = pinframe[1];
            poutframe[3] = pinframe[0]; 
        }
    }
}

and it is called by toARGB32:

QImage uldvideodec::toARGB32(QVideoFrame * fp){
   
    
    int width = fp->width();
    int height = fp->height();
    
    /* Calculate size for 4 bytes per pixel */
    QByteArray rgbBuffer(width*height*4,0);
    
    /* Get YUV frame */
    const uchar * pyuv = fp->bits();
    
    /* rgbBufferPointer */
    uchar * prgb = reinterpret_cast<uchar*>(rgbBuffer.data()); 
    
    
    /* Decode */
    qDebug()<<"Video Decoding:"<<fp->pixelFormat();
    m_yuvdecodedriver[fp->pixelFormat()].func(pyuv, prgb, width, height, m_yuvdecodedriver[fp->pixelFormat()].flag);        
    
    
    
    /* Create QImage */
    /* Encode in Qimage */
    QImage img = QImage(prgb,width,height,QImage::Format_ARGB32);
    
    return img;
    
}

[Julian Guarin]

@SGaist GLTextureHandle by the way

[SGaist]

You should check that you map call is successful before going further.

Then, since it's an OpenGL texture, IIRC you should access it as one using the handle to get back the data.

[Julian Guarin]

@SGaist Thank you, however, I use QVideoFrame::bits function which gives me a pointer to the video frame, Isn't this approach functional as well for Android? I mean, is very confusing... why the approach works on Mac/Win/Lnx, but it doesn't in Android? Why when saving the QImage to a PNG in a bytearray buffer a crash happens?

How would I access the GLTexture, out from a QVideoframe.. which is what I got....?

Thank you in advance.

[SGaist]

You can't access GPU memory (where the texture resides) like you do with CPU memory.

See here for how you can do it with OpenGL

[Julian Guarin]

@SGaist At the end my approach worked. The catch was to not get the decoded buffer inside a QImage in the QAbstractVideoFilter / QVideoFilterRunnable objects.

What I did was to pass a QByteArray as the container for the unsigned char * with the pixel data. In the run() function of the runnable:

QVideoFrame Uldfilterrunnable::run(QVideoFrame * input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags){
    
    Q_UNUSED(surfaceFormat);
    Q_UNUSED(flags);
    QByteArray ba;
    if (m_f->isGrabPending()){
        qDebug()<<input->map(QAbstractVideoBuffer::ReadOnly);
	ba = videodecoder.toARGB32ByteArray(input);
#ifdef ANDROID
	//debugVideoFrameInfo(input,surfaceFormat);
#endif	
	input->unmap();
	emit m_f->frameDecoded(ba,input->width(),input->height());
        qDebug()<<"Grab Done";
	m_f->m_grab.grab = false;
    }
    return *input;
    
}

The QBytearray was sent using a Queued connection to a slot in another QObject, and then I used the data in the array to create the QImage, and only then I issue the PNG serialization into a new QByteArray which I'm transmitting now :)

The slot was:

void UldWorker::pushImage(QByteArray imageChunk, int width, int height){
    /* image chunk */
    uchar * baidata = reinterpret_cast<uchar*>(imageChunk.data());
    /* Create QImage */
    QImage img = QImage(baidata, width, height, QImage::Format_ARGB32);
    /* Encode Image into PNG */
    QByteArray pngChunk;
    QBuffer pngChunkBuffer(&pngChunk);
    pngChunkBuffer.open(QIODevice::WriteOnly);
    //img = img.scaled(img.width()>>7, img.height()>>7);
    img.save(&pngChunkBuffer,"PNG");
    /*. Setup transmission.*/
    upload(pngChunk);
}

Uldworker is a QObject which happens to live in another Thread....

The connection I made was:

Uldfilterproject::Uldfilterproject(QObject* parent) : 
    QAbstractVideoFilter (parent),
    m_worker(new UldWorker())
    
{
    
    /* Change worker threads */
    m_worker->moveToThread(&m_thread);
    
    /* Connect end of decoding to image sending */
    connect(this,SIGNAL(frameDecoded(QByteArray,int,int)),m_worker,SLOT(pushImage(QByteArray,int,int)),Qt::QueuedConnection);
    

    m_thread.start();
}

[benlau]

@Julian-Guarin Do you find that the video frame is mirrored in Android?

The behaviour of the run() is quite strange in Android. I would think it is a blocked operation. But in fact it don't. I use OpenCV to convert BGR32 to ARGB and perform mirror. It is quite slow. It takes ~90ms for 1600x1200 image in Nexus 5. But It don't affect VideoOutput. It is still smooth.

Now I am thinking to use shader to convert BGR32.

[Julian Guarin]

@benlau mmmmm Quite strange if you do. But ain't my case......

Nevertheless are you using the map and unmap functions right? Better yet, do you have an example of the way u r using your function?. Is quite interesting, but there are no many examples of RunnableFilter::run functions out there....

[benlau]

@Julian-Guarin I am sorry. My information is not correct. The video output is still a bit choppy. But I have restricted to have only use a single thread for processing. So it look like smooth.

By the way, it is my another finding about the behaviour of QVideoFormat.

1. QVideoFrame->map() and unmap() already spent 50ms (tested in Nexus 5X). So video should be a bit choppy.

2. Don't use QVideoFrame->map(), and apply a custom shader. The speed is almost the same.

3. Use a custom shader and scale to half of the size. Then the time consumption will be dropped to around 25ms.