Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

How to pre-process QCamera frames using OpenCV before displaying on QCameraViewfinder



  • Hi All,

    I have been using QT5 for developing cross-platform applications for a short while although I am pretty good at C++ and stitching things together :)

    I have developed a small application to connect to webcam and display videos inspired by the below example:

    https://doc.qt.io/qt-5/qtmultimedia-multimediawidgets-camera-example.html

    I have bean trying to find a way to preprocess each frame before it is displayed by QCameraViewfinder (using opencv eg. apply edge detection filters and camera calibration ).

    So far I looked into pretty much many examples like below which turned out to be VERY slow:

    http://amin-ahmadi.com/2018/03/29/how-to-read-process-and-display-videos-using-qt-and-opencv/

    and also I have come across some other examples which mainly use qml which are NOT helpful cuz I am supposed to do everything in standard QT C++ as below:

    https://github.com/theshadowx/Qt_OpenCV/tree/master/QtQuick/CannyQml
    https://github.com/stephenquan/MyVideoFilterApp

    in which they use QAbstractVideoFilter QVideoFilterRunnable but I have no idea if they could be used by QCamer/QCameraViewfinder.

    But as I mentioned I am NOT supposed to use qml.

    I am just looking for a example describing how I can accomplish such task

    Much appreciate any clue and comment that could help me.

    Thank you all

    Mike


  • Lifetime Qt Champion

    Hi,

    How slow are they ?
    Did you consider moving the OpenCV processing part in its own thread and send a QImage to the GUI thread for displaying ?



  • I use timer with interval 100 but I get to see one 1fps. Really weird, I have to admit I might have not threading correctly.

    void MainWindow::on_startButton_clicked()
    {
    cout << "on_pushButton_clicked" << endl;
    // start camera
    ui->graphicsView->setScene(new QGraphicsScene(this));
    ui->graphicsView->scene()->addItem(&pixmap);
    ui->graphicsView->fitInView(&pixmap, Qt::KeepAspectRatio);
    video.open(0);

    thread = new QThread(this);
    timer = new QTimer();
    
    //Creating instance of Engine worker
    worker = new CaptureProcessVideoThread();
    worker->setVideoPixma(&video, &pixmap);
    worker->setSnapshotFolder("/home/mike/Pictures");
    worker->setCaptureListWidget(ui->captureListWidget);
    worker->setGraphicsView(ui->graphicsView);
    timer->setInterval(100);
    
    //Connecting Engine worker' foo slot to timer
    connect(timer, &QTimer::timeout, worker, &CaptureProcessVideoThread::run);
    connect(thread, &QThread::started, timer, static_cast<void (QTimer::*)(void)>(&QTimer::start));
    
    timer->moveToThread(thread);
    thread->start();
    

    }

    alt text

    Even if you can comment on the multi-threading that would help a lot.

    Thank you


  • Lifetime Qt Champion

    Can you show the CaptureProcessVideoThread class content ?



  • @SGaist said in How to pre-process QCamera frames using OpenCV before displaying on QCameraViewfinder:

    CaptureProcessVideoThread

    Sure ....

    **captureprocessvieothread.h**
    #ifndef CAPTUREPROCESSVIDEOTHREAD_H
    #define CAPTUREPROCESSVIDEOTHREAD_H
    
    #include <QThread>
    #include <opencv/cv.hpp>
    #include <QGraphicsPixmapItem>
    #include <QListWidget>
    #include <QGraphicsView>
    using namespace cv;
    
    class CaptureProcessVideoThread : public QObject
    {
    public slots:
        void run();
        void captureSnapshot();
    
    public:
        //CaptureProcessVideoThread(QGraphicsPixmapItem *pixmap, VideoCapture *video);
        CaptureProcessVideoThread();
        void setVideoPixma(VideoCapture *video, QGraphicsPixmapItem *pixmap);
        void setSnapshotFolder(QString snapshotFolder);
        void setCaptureListWidget(QListWidget*);
        void setGraphicsView(QGraphicsView*);
    private:
        QGraphicsView *m_graphicsView;
        QGraphicsPixmapItem *m_pixmap;
        VideoCapture *m_video;
        QString m_snapshotFolder;
        bool m_takeSnapshot = false;
        QListWidget *m_captureListWideget;
    };
    
    #endif // CAPTUREPROCESSVIDEOTHREAD_H
    
    
    **captureprocessvieothread.cpp**
    
    #include "captureprocessvideothread.h"
    #include<QTextStream>
    #include<QFileInfo>
    
    #include <utils/settings.h>
    #include <utils/utils.h>
    using namespace cv;
    using namespace std;
    
    CaptureProcessVideoThread::CaptureProcessVideoThread()
        {
    
    }
    
    void CaptureProcessVideoThread::setVideoPixma(VideoCapture *video, QGraphicsPixmapItem *pixmap){
        m_video = video;
        m_pixmap = pixmap;
    }
    void CaptureProcessVideoThread::setSnapshotFolder(QString snapshotFolder){
        m_snapshotFolder = snapshotFolder;
    }
    
    void CaptureProcessVideoThread::setCaptureListWidget(QListWidget *captureListWideget){
        m_captureListWideget = captureListWideget;
    }
    
    void CaptureProcessVideoThread::run(){
        Mat frame, frameToSave;
        QTextStream cout(stdout);
    
        // index for snapshots
        static int lastSnapIndex = 0;
    
        if (m_video->isOpened()){
            *m_video >> frame;
            *m_video >> frameToSave;
    
            // try to find chessboard corners
            // if found draw them
            int chessBoardFlags = CALIB_CB_ADAPTIVE_THRESH | CALIB_CB_NORMALIZE_IMAGE;
            vector<Point2f> pointBuf;
            Settings s;
            //        s.boardSize = Size(19, 11);
            s = getCalibrationSettings();
            bool found = findChessboardCorners( frame, s.boardSize, pointBuf, chessBoardFlags);
            if (found){
                drawChessboardCorners( frame, s.boardSize, Mat(pointBuf), found );
            }
    
            // create QImage from frame
            QImage qimg(frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888);        
            QImage qimgToSave(frameToSave.data, frameToSave.cols, frameToSave.rows, frameToSave.step, QImage::Format_RGB888);
    
            // update the Graphvis the view with the new frame
            m_pixmap->setPixmap( QPixmap::fromImage(qimg.rgbSwapped()) );
            m_graphicsView->fitInView(m_pixmap, Qt::KeepAspectRatio);
    
            // take snapshot if asked for
            if(m_takeSnapshot){
                QString fileName = m_snapshotFolder + QString("/Capture_%1d.png").arg(++lastSnapIndex, 5, 10, QChar('0'));
                QFileInfo fileInfo(fileName);            
                qimgToSave.save(fileName);
    
                // add the snapshot to list of snapshots
                m_captureListWideget->addItem(new QListWidgetItem( QIcon(fileName), fileInfo.fileName()));
    
                m_takeSnapshot = false;
            }
        }
    }
    void CaptureProcessVideoThread::setGraphicsView(QGraphicsView *graphicsView){
        m_graphicsView = graphicsView;
    }
    
    void CaptureProcessVideoThread::captureSnapshot(){
        m_takeSnapshot = true;
    }
    

  • Lifetime Qt Champion

    You should avoid re-creating the pixmap, once you have it with the correct size, just memcpy the content of the processed frame into place.

    You should also avoid calling fitInView from that external thread. GUI operation should only happen in the GUI thread. You're lucky it doesn't crash.

    Same goes for the snapshot handling.

    Did you try to get some performance number from your run method ?



  • so .... you mean sth like: QPixmap::loadFromData

    I tried this but no luck so far....

            QByteArray arr;
            QBuffer buffer(&arr);
            buffer.open(QIODevice::WriteOnly);
            qimg.save(&buffer);
            m_pixmap->pixmap().loadFromData(arr);
    
    

    Looks like I have done correctly before :

    https://www.qtcentre.org/threads/56871-QGraphicsPixmapItem-repaint-immediately

    No errors ... but I don't see anything...



  • The source code for setPixmap is like below:

    void QGraphicsPixmapItem::setPixmap(const QPixmap &pixmap)
    {
        Q_D(QGraphicsPixmapItem);
        prepareGeometryChange();
        d->pixmap = pixmap;
        d->hasShape = false;
        update();
    }
    

    from here: https://code.woboq.org/qt5/qtbase/src/widgets/graphicsview/qgraphicsitem.cpp.html#9691

    The only way seems to keep access and update d->pixmap directly using a memcopy sth like that just to avoid changing the location of the image to be viewed in the memory.

    am I correct?

    QPixmap QGraphicsPixmapItem::pixmap() const
    {
        Q_D(const QGraphicsPixmapItem);
        return d->pixmap;
    }
    

  • Lifetime Qt Champion

    It's the idea. Avoid re-creating objects if possible. By the way, can't you get OpenCV to directly give you image data in RGB rather than BGR ? That would avoid a copy operation.



  • @mikeitexpert thanks mike, can you share please the: #include <utils/settings.h>
    #include <utils/utils.h> ?


Log in to reply