QImage manipulation crashes on android

Julian Guarin

The following code:

QVideoFrame Uldfilterrunnable::run(QVideoFrame * input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags){
    
    Q_UNUSED(surfaceFormat);
    Q_UNUSED(flags);
    
    if (m_f->isGrabPending()){
        QRect selectionRectangle = m_f -> grabSelectionRect();
        input->map(QAbstractVideoBuffer::ReadOnly);
        QImage vimgtotal = videodecoder.toARGB32(input);
        input->unmap();
        QImage vimgselection = selectionRectangle.isNull() ? vimgtotal : vimgtotal.copy(selectionRectangle);
        m_f->grabDone(vimgselection); //****OFFENDING CALL******
    }
    
    return *input;
    
}

 void Uldfilterproject::grabDone(QImage &img){
        m_grab.grab = false;
        m_worker -> imagePush(img); //****OFFENDING CALL******
}
void UldWorker::imagePush(QImage i){
    
    if (i.isNull()){
        qDebug()<<"Null QImage";
        return;
    }
    /* Sometime in the near future set a QJson Object Along with PNG file */
    /* Convert image to buffered RAM file */
    QByteArray pngChunk;
    QBuffer pngChunkBuffer(&pngChunk);
    pngChunkBuffer.open(QIODevice::WriteOnly);
    i.save(&pngChunkBuffer,"PNG"); ****OFFENDING LINE******: ANY QIMAGE READ WRITE WOULD CRASH ON ANDROID.
    qDebug()<<"PUSH";
    m_qi.enqueue(pngChunk);
    qDebug()<<"QUEUED";
    return;
}

Thing is both Uldfilterrunabble::run and Uldworker::imagePush both functions run in different QThreads..... However this code works on on Mac OSX, Windows and Linux, but it misbehaves in Android, any attempt to Read Write QImage, would crash. Wonder if anyone has ever had this crash?

Julian Guarin

@SGaist At the end my approach worked. The catch was to not get the decoded buffer inside a QImage in the QAbstractVideoFilter / QVideoFilterRunnable objects.

What I did was to pass a QByteArray as the container for the unsigned char * with the pixel data. In the run() function of the runnable:

QVideoFrame Uldfilterrunnable::run(QVideoFrame * input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags){
    
    Q_UNUSED(surfaceFormat);
    Q_UNUSED(flags);
    QByteArray ba;
    if (m_f->isGrabPending()){
        qDebug()<<input->map(QAbstractVideoBuffer::ReadOnly);
	ba = videodecoder.toARGB32ByteArray(input);
#ifdef ANDROID
	//debugVideoFrameInfo(input,surfaceFormat);
#endif	
	input->unmap();
	emit m_f->frameDecoded(ba,input->width(),input->height());
        qDebug()<<"Grab Done";
	m_f->m_grab.grab = false;
    }
    return *input;
    
}

The QBytearray was sent using a Queued connection to a slot in another QObject, and then I used the data in the array to create the QImage, and only then I issue the PNG serialization into a new QByteArray which I'm transmitting now :)

The slot was:

void UldWorker::pushImage(QByteArray imageChunk, int width, int height){
    /* image chunk */
    uchar * baidata = reinterpret_cast<uchar*>(imageChunk.data());
    /* Create QImage */
    QImage img = QImage(baidata, width, height, QImage::Format_ARGB32);
    /* Encode Image into PNG */
    QByteArray pngChunk;
    QBuffer pngChunkBuffer(&pngChunk);
    pngChunkBuffer.open(QIODevice::WriteOnly);
    //img = img.scaled(img.width()>>7, img.height()>>7);
    img.save(&pngChunkBuffer,"PNG");
    /*. Setup transmission.*/
    upload(pngChunk);
}

Uldworker is a QObject which happens to live in another Thread....

The connection I made was:


Uldfilterproject::Uldfilterproject(QObject* parent) : 
    QAbstractVideoFilter (parent),
    m_worker(new UldWorker())
    
{
    
    /* Change worker threads */
    m_worker->moveToThread(&m_thread);
    
    /* Connect end of decoding to image sending */
    connect(this,SIGNAL(frameDecoded(QByteArray,int,int)),m_worker,SLOT(pushImage(QByteArray,int,int)),Qt::QueuedConnection);
    

    m_thread.start();
}

mvuori

Unfortunately, things that one should not use and which should not work, do sometimes work - until they stop working when the environment changes...

I guess that you should accept that this is not the way to implement things even though it works on 75 % of environments. Try using signals & slots for passing the image.

Julian Guarin

Hehehe the Qt illusion.... I guess.....

SGaist

Hi,

Did you check what type of data you got with your QVideoFrame ? It might be an OpenGL texture, is that a supported case by your videodecoder object ?

Julian Guarin

@SGaist Thank you.

I will check, nevertheless I'm checking the pixel format... which is BGR32 in the android run it returns (QVideoFrame::Format_BGR32), on the Mac it returns (QVideoFrame::Format_NV12).

So the decoding function is for Android case is:

void qt_convert_BGR32_to_ARGB32(const uchar *pinframe, uchar *poutframe, int width, int height, int flags){
    Q_UNUSED(flags);
    for (int i = 0; i < height; i++){
        for (int j = 0; j < width; j ++, pinframe +=4, poutframe +=4){
            poutframe[0] = pinframe[3];
            poutframe[1] = pinframe[2];
            poutframe[2] = pinframe[1];
            poutframe[3] = pinframe[0]; 
        }
    }
}

and it is called by toARGB32:

QImage uldvideodec::toARGB32(QVideoFrame * fp){
   
    
    int width = fp->width();
    int height = fp->height();
    
    /* Calculate size for 4 bytes per pixel */
    QByteArray rgbBuffer(width*height*4,0);
    
    /* Get YUV frame */
    const uchar * pyuv = fp->bits();
    
    /* rgbBufferPointer */
    uchar * prgb = reinterpret_cast<uchar*>(rgbBuffer.data()); 
    
    
    /* Decode */
    qDebug()<<"Video Decoding:"<<fp->pixelFormat();
    m_yuvdecodedriver[fp->pixelFormat()].func(pyuv, prgb, width, height, m_yuvdecodedriver[fp->pixelFormat()].flag);        
    
    
    
    /* Create QImage */
    /* Encode in Qimage */
    QImage img = QImage(prgb,width,height,QImage::Format_ARGB32);
    
    return img;
    
}

Julian Guarin

@SGaist GLTextureHandle by the way

SGaist

You should check that you map call is successful before going further.

Then, since it's an OpenGL texture, IIRC you should access it as one using the handle to get back the data.

Julian Guarin

@SGaist Thank you, however, I use QVideoFrame::bits function which gives me a pointer to the video frame, Isn't this approach functional as well for Android? I mean, is very confusing... why the approach works on Mac/Win/Lnx, but it doesn't in Android? Why when saving the QImage to a PNG in a bytearray buffer a crash happens?

How would I access the GLTexture, out from a QVideoframe.. which is what I got....?

Thank you in advance.

SGaist

You can't access GPU memory (where the texture resides) like you do with CPU memory.

See here for how you can do it with OpenGL

Julian Guarin

@SGaist At the end my approach worked. The catch was to not get the decoded buffer inside a QImage in the QAbstractVideoFilter / QVideoFilterRunnable objects.

What I did was to pass a QByteArray as the container for the unsigned char * with the pixel data. In the run() function of the runnable:

QVideoFrame Uldfilterrunnable::run(QVideoFrame * input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags){
    
    Q_UNUSED(surfaceFormat);
    Q_UNUSED(flags);
    QByteArray ba;
    if (m_f->isGrabPending()){
        qDebug()<<input->map(QAbstractVideoBuffer::ReadOnly);
	ba = videodecoder.toARGB32ByteArray(input);
#ifdef ANDROID
	//debugVideoFrameInfo(input,surfaceFormat);
#endif	
	input->unmap();
	emit m_f->frameDecoded(ba,input->width(),input->height());
        qDebug()<<"Grab Done";
	m_f->m_grab.grab = false;
    }
    return *input;
    
}

The QBytearray was sent using a Queued connection to a slot in another QObject, and then I used the data in the array to create the QImage, and only then I issue the PNG serialization into a new QByteArray which I'm transmitting now :)

The slot was:

void UldWorker::pushImage(QByteArray imageChunk, int width, int height){
    /* image chunk */
    uchar * baidata = reinterpret_cast<uchar*>(imageChunk.data());
    /* Create QImage */
    QImage img = QImage(baidata, width, height, QImage::Format_ARGB32);
    /* Encode Image into PNG */
    QByteArray pngChunk;
    QBuffer pngChunkBuffer(&pngChunk);
    pngChunkBuffer.open(QIODevice::WriteOnly);
    //img = img.scaled(img.width()>>7, img.height()>>7);
    img.save(&pngChunkBuffer,"PNG");
    /*. Setup transmission.*/
    upload(pngChunk);
}

Uldworker is a QObject which happens to live in another Thread....

The connection I made was:


Uldfilterproject::Uldfilterproject(QObject* parent) : 
    QAbstractVideoFilter (parent),
    m_worker(new UldWorker())
    
{
    
    /* Change worker threads */
    m_worker->moveToThread(&m_thread);
    
    /* Connect end of decoding to image sending */
    connect(this,SIGNAL(frameDecoded(QByteArray,int,int)),m_worker,SLOT(pushImage(QByteArray,int,int)),Qt::QueuedConnection);
    

    m_thread.start();
}

benlau

@Julian-Guarin Do you find that the video frame is mirrored in Android?

The behaviour of the run() is quite strange in Android. I would think it is a blocked operation. But in fact it don't. I use OpenCV to convert BGR32 to ARGB and perform mirror. It is quite slow. It takes ~90ms for 1600x1200 image in Nexus 5. But It don't affect VideoOutput. It is still smooth.

Now I am thinking to use shader to convert BGR32.

Julian Guarin

@benlau mmmmm Quite strange if you do. But ain't my case......

Nevertheless are you using the map and unmap functions right? Better yet, do you have an example of the way u r using your function?. Is quite interesting, but there are no many examples of RunnableFilter::run functions out there....

benlau

@Julian-Guarin I am sorry. My information is not correct. The video output is still a bit choppy. But I have restricted to have only use a single thread for processing. So it look like smooth.

By the way, it is my another finding about the behaviour of QVideoFormat.

QVideoFrame->map() and unmap() already spent 50ms (tested in Nexus 5X). So video should be a bit choppy.
Don't use QVideoFrame->map(), and apply a custom shader. The speed is almost the same.
Use a custom shader and scale to half of the size. Then the time consumption will be dropped to around 25ms.