How to pre-process QCamera frames using OpenCV before displaying on QCameraViewfinder

mikeitexpert

Hi All,

I have been using QT5 for developing cross-platform applications for a short while although I am pretty good at C++ and stitching things together :)

I have developed a small application to connect to webcam and display videos inspired by the below example:

https://doc.qt.io/qt-5/qtmultimedia-multimediawidgets-camera-example.html

I have bean trying to find a way to preprocess each frame before it is displayed by QCameraViewfinder (using opencv eg. apply edge detection filters and camera calibration ).

So far I looked into pretty much many examples like below which turned out to be VERY slow:

http://amin-ahmadi.com/2018/03/29/how-to-read-process-and-display-videos-using-qt-and-opencv/

and also I have come across some other examples which mainly use qml which are NOT helpful cuz I am supposed to do everything in standard QT C++ as below:

https://github.com/theshadowx/Qt_OpenCV/tree/master/QtQuick/CannyQml
https://github.com/stephenquan/MyVideoFilterApp

in which they use QAbstractVideoFilter QVideoFilterRunnable but I have no idea if they could be used by QCamer/QCameraViewfinder.

But as I mentioned I am NOT supposed to use qml.

I am just looking for a example describing how I can accomplish such task

Much appreciate any clue and comment that could help me.

Thank you all

Mike

SGaist

Hi,

How slow are they ?
Did you consider moving the OpenCV processing part in its own thread and send a QImage to the GUI thread for displaying ?

mikeitexpert

I use timer with interval 100 but I get to see one 1fps. Really weird, I have to admit I might have not threading correctly.

void MainWindow::on_startButton_clicked()
{
cout << "on_pushButton_clicked" << endl;
// start camera
ui->graphicsView->setScene(new QGraphicsScene(this));
ui->graphicsView->scene()->addItem(&pixmap);
ui->graphicsView->fitInView(&pixmap, Qt::KeepAspectRatio);
video.open(0);

thread = new QThread(this);
timer = new QTimer();

//Creating instance of Engine worker
worker = new CaptureProcessVideoThread();
worker->setVideoPixma(&video, &pixmap);
worker->setSnapshotFolder("/home/mike/Pictures");
worker->setCaptureListWidget(ui->captureListWidget);
worker->setGraphicsView(ui->graphicsView);
timer->setInterval(100);

//Connecting Engine worker' foo slot to timer
connect(timer, &QTimer::timeout, worker, &CaptureProcessVideoThread::run);
connect(thread, &QThread::started, timer, static_cast<void (QTimer::*)(void)>(&QTimer::start));

timer->moveToThread(thread);
thread->start();

}

alt text

Even if you can comment on the multi-threading that would help a lot.

Thank you

SGaist

Can you show the CaptureProcessVideoThread class content ?

mikeitexpert

@SGaist said in How to pre-process QCamera frames using OpenCV before displaying on QCameraViewfinder:

CaptureProcessVideoThread

Sure ....

**captureprocessvieothread.h**
#ifndef CAPTUREPROCESSVIDEOTHREAD_H
#define CAPTUREPROCESSVIDEOTHREAD_H

#include <QThread>
#include <opencv/cv.hpp>
#include <QGraphicsPixmapItem>
#include <QListWidget>
#include <QGraphicsView>
using namespace cv;

class CaptureProcessVideoThread : public QObject
{
public slots:
    void run();
    void captureSnapshot();

public:
    //CaptureProcessVideoThread(QGraphicsPixmapItem *pixmap, VideoCapture *video);
    CaptureProcessVideoThread();
    void setVideoPixma(VideoCapture *video, QGraphicsPixmapItem *pixmap);
    void setSnapshotFolder(QString snapshotFolder);
    void setCaptureListWidget(QListWidget*);
    void setGraphicsView(QGraphicsView*);
private:
    QGraphicsView *m_graphicsView;
    QGraphicsPixmapItem *m_pixmap;
    VideoCapture *m_video;
    QString m_snapshotFolder;
    bool m_takeSnapshot = false;
    QListWidget *m_captureListWideget;
};

#endif // CAPTUREPROCESSVIDEOTHREAD_H


**captureprocessvieothread.cpp**

#include "captureprocessvideothread.h"
#include<QTextStream>
#include<QFileInfo>

#include <utils/settings.h>
#include <utils/utils.h>
using namespace cv;
using namespace std;

CaptureProcessVideoThread::CaptureProcessVideoThread()
    {

}

void CaptureProcessVideoThread::setVideoPixma(VideoCapture *video, QGraphicsPixmapItem *pixmap){
    m_video = video;
    m_pixmap = pixmap;
}
void CaptureProcessVideoThread::setSnapshotFolder(QString snapshotFolder){
    m_snapshotFolder = snapshotFolder;
}

void CaptureProcessVideoThread::setCaptureListWidget(QListWidget *captureListWideget){
    m_captureListWideget = captureListWideget;
}

void CaptureProcessVideoThread::run(){
    Mat frame, frameToSave;
    QTextStream cout(stdout);

    // index for snapshots
    static int lastSnapIndex = 0;

    if (m_video->isOpened()){
        *m_video >> frame;
        *m_video >> frameToSave;

        // try to find chessboard corners
        // if found draw them
        int chessBoardFlags = CALIB_CB_ADAPTIVE_THRESH | CALIB_CB_NORMALIZE_IMAGE;
        vector<Point2f> pointBuf;
        Settings s;
        //        s.boardSize = Size(19, 11);
        s = getCalibrationSettings();
        bool found = findChessboardCorners( frame, s.boardSize, pointBuf, chessBoardFlags);
        if (found){
            drawChessboardCorners( frame, s.boardSize, Mat(pointBuf), found );
        }

        // create QImage from frame
        QImage qimg(frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888);        
        QImage qimgToSave(frameToSave.data, frameToSave.cols, frameToSave.rows, frameToSave.step, QImage::Format_RGB888);

        // update the Graphvis the view with the new frame
        m_pixmap->setPixmap( QPixmap::fromImage(qimg.rgbSwapped()) );
        m_graphicsView->fitInView(m_pixmap, Qt::KeepAspectRatio);

        // take snapshot if asked for
        if(m_takeSnapshot){
            QString fileName = m_snapshotFolder + QString("/Capture_%1d.png").arg(++lastSnapIndex, 5, 10, QChar('0'));
            QFileInfo fileInfo(fileName);            
            qimgToSave.save(fileName);

            // add the snapshot to list of snapshots
            m_captureListWideget->addItem(new QListWidgetItem( QIcon(fileName), fileInfo.fileName()));

            m_takeSnapshot = false;
        }
    }
}
void CaptureProcessVideoThread::setGraphicsView(QGraphicsView *graphicsView){
    m_graphicsView = graphicsView;
}

void CaptureProcessVideoThread::captureSnapshot(){
    m_takeSnapshot = true;
}

SGaist

You should avoid re-creating the pixmap, once you have it with the correct size, just memcpy the content of the processed frame into place.

You should also avoid calling fitInView from that external thread. GUI operation should only happen in the GUI thread. You're lucky it doesn't crash.

Same goes for the snapshot handling.

Did you try to get some performance number from your run method ?

mikeitexpert

so .... you mean sth like: QPixmap::loadFromData

I tried this but no luck so far....

        QByteArray arr;
        QBuffer buffer(&arr);
        buffer.open(QIODevice::WriteOnly);
        qimg.save(&buffer);
        m_pixmap->pixmap().loadFromData(arr);

Looks like I have done correctly before :

https://www.qtcentre.org/threads/56871-QGraphicsPixmapItem-repaint-immediately

No errors ... but I don't see anything...

mikeitexpert

The source code for setPixmap is like below:

void QGraphicsPixmapItem::setPixmap(const QPixmap &pixmap)
{
    Q_D(QGraphicsPixmapItem);
    prepareGeometryChange();
    d->pixmap = pixmap;
    d->hasShape = false;
    update();
}

from here: https://code.woboq.org/qt5/qtbase/src/widgets/graphicsview/qgraphicsitem.cpp.html#9691

The only way seems to keep access and update d->pixmap directly using a memcopy sth like that just to avoid changing the location of the image to be viewed in the memory.

am I correct?

QPixmap QGraphicsPixmapItem::pixmap() const
{
    Q_D(const QGraphicsPixmapItem);
    return d->pixmap;
}

SGaist

It's the idea. Avoid re-creating objects if possible. By the way, can't you get OpenCV to directly give you image data in RGB rather than BGR ? That would avoid a copy operation.

Maric

@mikeitexpert thanks mike, can you share please the: #include <utils/settings.h>
#include <utils/utils.h> ?