QImage manipulation crashes on android



  • The following code:

    QVideoFrame Uldfilterrunnable::run(QVideoFrame * input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags){
        
        Q_UNUSED(surfaceFormat);
        Q_UNUSED(flags);
        
        if (m_f->isGrabPending()){
            QRect selectionRectangle = m_f -> grabSelectionRect();
            input->map(QAbstractVideoBuffer::ReadOnly);
            QImage vimgtotal = videodecoder.toARGB32(input);
            input->unmap();
            QImage vimgselection = selectionRectangle.isNull() ? vimgtotal : vimgtotal.copy(selectionRectangle);
            m_f->grabDone(vimgselection); //****OFFENDING CALL******
        }
        
        return *input;
        
    }
    
     void Uldfilterproject::grabDone(QImage &img){
            m_grab.grab = false;
            m_worker -> imagePush(img); //****OFFENDING CALL******
    }
    void UldWorker::imagePush(QImage i){
        
        if (i.isNull()){
            qDebug()<<"Null QImage";
            return;
        }
        /* Sometime in the near future set a QJson Object Along with PNG file */
        /* Convert image to buffered RAM file */
        QByteArray pngChunk;
        QBuffer pngChunkBuffer(&pngChunk);
        pngChunkBuffer.open(QIODevice::WriteOnly);
        i.save(&pngChunkBuffer,"PNG"); ****OFFENDING LINE******: ANY QIMAGE READ WRITE WOULD CRASH ON ANDROID.
        qDebug()<<"PUSH";
        m_qi.enqueue(pngChunk);
        qDebug()<<"QUEUED";
        return;
    }
    
    
    

    Thing is both Uldfilterrunabble::run and Uldworker::imagePush both functions run in different QThreads..... However this code works on on Mac OSX, Windows and Linux, but it misbehaves in Android, any attempt to Read Write QImage, would crash. Wonder if anyone has ever had this crash?



  • Unfortunately, things that one should not use and which should not work, do sometimes work - until they stop working when the environment changes...

    I guess that you should accept that this is not the way to implement things even though it works on 75 % of environments. Try using signals & slots for passing the image.



  • Hehehe the Qt illusion.... I guess.....


  • Lifetime Qt Champion

    Hi,

    Did you check what type of data you got with your QVideoFrame ? It might be an OpenGL texture, is that a supported case by your videodecoder object ?



  • @SGaist Thank you.

    I will check, nevertheless I'm checking the pixel format... which is BGR32 in the android run it returns (QVideoFrame::Format_BGR32), on the Mac it returns (QVideoFrame::Format_NV12).

    So the decoding function is for Android case is:

    void qt_convert_BGR32_to_ARGB32(const uchar *pinframe, uchar *poutframe, int width, int height, int flags){
        Q_UNUSED(flags);
        for (int i = 0; i < height; i++){
            for (int j = 0; j < width; j ++, pinframe +=4, poutframe +=4){
                poutframe[0] = pinframe[3];
                poutframe[1] = pinframe[2];
                poutframe[2] = pinframe[1];
                poutframe[3] = pinframe[0]; 
            }
        }
    }
    

    and it is called by toARGB32:

    QImage uldvideodec::toARGB32(QVideoFrame * fp){
       
        
        int width = fp->width();
        int height = fp->height();
        
        /* Calculate size for 4 bytes per pixel */
        QByteArray rgbBuffer(width*height*4,0);
        
        /* Get YUV frame */
        const uchar * pyuv = fp->bits();
        
        /* rgbBufferPointer */
        uchar * prgb = reinterpret_cast<uchar*>(rgbBuffer.data()); 
        
        
        /* Decode */
        qDebug()<<"Video Decoding:"<<fp->pixelFormat();
        m_yuvdecodedriver[fp->pixelFormat()].func(pyuv, prgb, width, height, m_yuvdecodedriver[fp->pixelFormat()].flag);        
        
        
        
        /* Create QImage */
        /* Encode in Qimage */
        QImage img = QImage(prgb,width,height,QImage::Format_ARGB32);
        
        return img;
        
    }
    


  • @SGaist GLTextureHandle by the way


  • Lifetime Qt Champion

    You should check that you map call is successful before going further.

    Then, since it's an OpenGL texture, IIRC you should access it as one using the handle to get back the data.



  • @SGaist Thank you, however, I use QVideoFrame::bits function which gives me a pointer to the video frame, Isn't this approach functional as well for Android? I mean, is very confusing... why the approach works on Mac/Win/Lnx, but it doesn't in Android? Why when saving the QImage to a PNG in a bytearray buffer a crash happens?

    How would I access the GLTexture, out from a QVideoframe.. which is what I got....?

    Thank you in advance.


  • Lifetime Qt Champion

    You can't access GPU memory (where the texture resides) like you do with CPU memory.

    See here for how you can do it with OpenGL



  • @SGaist At the end my approach worked. The catch was to not get the decoded buffer inside a QImage in the QAbstractVideoFilter / QVideoFilterRunnable objects.

    What I did was to pass a QByteArray as the container for the unsigned char * with the pixel data. In the run() function of the runnable:

    QVideoFrame Uldfilterrunnable::run(QVideoFrame * input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags){
        
        Q_UNUSED(surfaceFormat);
        Q_UNUSED(flags);
        QByteArray ba;
        if (m_f->isGrabPending()){
            qDebug()<<input->map(QAbstractVideoBuffer::ReadOnly);
    	ba = videodecoder.toARGB32ByteArray(input);
    #ifdef ANDROID
    	//debugVideoFrameInfo(input,surfaceFormat);
    #endif	
    	input->unmap();
    	emit m_f->frameDecoded(ba,input->width(),input->height());
            qDebug()<<"Grab Done";
    	m_f->m_grab.grab = false;
        }
        return *input;
        
    }
    

    The QBytearray was sent using a Queued connection to a slot in another QObject, and then I used the data in the array to create the QImage, and only then I issue the PNG serialization into a new QByteArray which I'm transmitting now :)

    The slot was:

    void UldWorker::pushImage(QByteArray imageChunk, int width, int height){
        /* image chunk */
        uchar * baidata = reinterpret_cast<uchar*>(imageChunk.data());
        /* Create QImage */
        QImage img = QImage(baidata, width, height, QImage::Format_ARGB32);
        /* Encode Image into PNG */
        QByteArray pngChunk;
        QBuffer pngChunkBuffer(&pngChunk);
        pngChunkBuffer.open(QIODevice::WriteOnly);
        //img = img.scaled(img.width()>>7, img.height()>>7);
        img.save(&pngChunkBuffer,"PNG");
        /*. Setup transmission.*/
        upload(pngChunk);
    }
    

    Uldworker is a QObject which happens to live in another Thread....

    The connection I made was:

    
    Uldfilterproject::Uldfilterproject(QObject* parent) : 
        QAbstractVideoFilter (parent),
        m_worker(new UldWorker())
        
    {
        
        /* Change worker threads */
        m_worker->moveToThread(&m_thread);
        
        /* Connect end of decoding to image sending */
        connect(this,SIGNAL(frameDecoded(QByteArray,int,int)),m_worker,SLOT(pushImage(QByteArray,int,int)),Qt::QueuedConnection);
        
    
        m_thread.start();
    }
    
    

  • Qt Champions 2016

    @Julian-Guarin Do you find that the video frame is mirrored in Android?

    The behaviour of the run() is quite strange in Android. I would think it is a blocked operation. But in fact it don't. I use OpenCV to convert BGR32 to ARGB and perform mirror. It is quite slow. It takes ~90ms for 1600x1200 image in Nexus 5. But It don't affect VideoOutput. It is still smooth.

    Now I am thinking to use shader to convert BGR32.



  • @benlau mmmmm Quite strange if you do. But ain't my case......

    Nevertheless are you using the map and unmap functions right? Better yet, do you have an example of the way u r using your function?. Is quite interesting, but there are no many examples of RunnableFilter::run functions out there....


  • Qt Champions 2016

    @Julian-Guarin I am sorry. My information is not correct. The video output is still a bit choppy. But I have restricted to have only use a single thread for processing. So it look like smooth.

    By the way, it is my another finding about the behaviour of QVideoFormat.

    1. QVideoFrame->map() and unmap() already spent 50ms (tested in Nexus 5X). So video should be a bit choppy.

    2. Don't use QVideoFrame->map(), and apply a custom shader. The speed is almost the same.

    3. Use a custom shader and scale to half of the size. Then the time consumption will be dropped to around 25ms.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.