Unsolved How to pre-process QCamera frames using OpenCV before displaying on QCameraViewfinder
-
Hi All,
I have been using QT5 for developing cross-platform applications for a short while although I am pretty good at C++ and stitching things together :)
I have developed a small application to connect to webcam and display videos inspired by the below example:
https://doc.qt.io/qt-5/qtmultimedia-multimediawidgets-camera-example.html
I have bean trying to find a way to preprocess each frame before it is displayed by QCameraViewfinder (using opencv eg. apply edge detection filters and camera calibration ).
So far I looked into pretty much many examples like below which turned out to be VERY slow:
http://amin-ahmadi.com/2018/03/29/how-to-read-process-and-display-videos-using-qt-and-opencv/
and also I have come across some other examples which mainly use qml which are NOT helpful cuz I am supposed to do everything in standard QT C++ as below:
https://github.com/theshadowx/Qt_OpenCV/tree/master/QtQuick/CannyQml
https://github.com/stephenquan/MyVideoFilterAppin which they use QAbstractVideoFilter QVideoFilterRunnable but I have no idea if they could be used by QCamer/QCameraViewfinder.
But as I mentioned I am NOT supposed to use qml.
I am just looking for a example describing how I can accomplish such task
Much appreciate any clue and comment that could help me.
Thank you all
Mike
-
Hi,
How slow are they ?
Did you consider moving the OpenCV processing part in its own thread and send a QImage to the GUI thread for displaying ? -
I use timer with interval 100 but I get to see one 1fps. Really weird, I have to admit I might have not threading correctly.
void MainWindow::on_startButton_clicked()
{
cout << "on_pushButton_clicked" << endl;
// start camera
ui->graphicsView->setScene(new QGraphicsScene(this));
ui->graphicsView->scene()->addItem(&pixmap);
ui->graphicsView->fitInView(&pixmap, Qt::KeepAspectRatio);
video.open(0);thread = new QThread(this); timer = new QTimer(); //Creating instance of Engine worker worker = new CaptureProcessVideoThread(); worker->setVideoPixma(&video, &pixmap); worker->setSnapshotFolder("/home/mike/Pictures"); worker->setCaptureListWidget(ui->captureListWidget); worker->setGraphicsView(ui->graphicsView); timer->setInterval(100); //Connecting Engine worker' foo slot to timer connect(timer, &QTimer::timeout, worker, &CaptureProcessVideoThread::run); connect(thread, &QThread::started, timer, static_cast<void (QTimer::*)(void)>(&QTimer::start)); timer->moveToThread(thread); thread->start();
}
Even if you can comment on the multi-threading that would help a lot.
Thank you
-
Can you show the CaptureProcessVideoThread class content ?
-
@SGaist said in How to pre-process QCamera frames using OpenCV before displaying on QCameraViewfinder:
CaptureProcessVideoThread
Sure ....
**captureprocessvieothread.h** #ifndef CAPTUREPROCESSVIDEOTHREAD_H #define CAPTUREPROCESSVIDEOTHREAD_H #include <QThread> #include <opencv/cv.hpp> #include <QGraphicsPixmapItem> #include <QListWidget> #include <QGraphicsView> using namespace cv; class CaptureProcessVideoThread : public QObject { public slots: void run(); void captureSnapshot(); public: //CaptureProcessVideoThread(QGraphicsPixmapItem *pixmap, VideoCapture *video); CaptureProcessVideoThread(); void setVideoPixma(VideoCapture *video, QGraphicsPixmapItem *pixmap); void setSnapshotFolder(QString snapshotFolder); void setCaptureListWidget(QListWidget*); void setGraphicsView(QGraphicsView*); private: QGraphicsView *m_graphicsView; QGraphicsPixmapItem *m_pixmap; VideoCapture *m_video; QString m_snapshotFolder; bool m_takeSnapshot = false; QListWidget *m_captureListWideget; }; #endif // CAPTUREPROCESSVIDEOTHREAD_H **captureprocessvieothread.cpp** #include "captureprocessvideothread.h" #include<QTextStream> #include<QFileInfo> #include <utils/settings.h> #include <utils/utils.h> using namespace cv; using namespace std; CaptureProcessVideoThread::CaptureProcessVideoThread() { } void CaptureProcessVideoThread::setVideoPixma(VideoCapture *video, QGraphicsPixmapItem *pixmap){ m_video = video; m_pixmap = pixmap; } void CaptureProcessVideoThread::setSnapshotFolder(QString snapshotFolder){ m_snapshotFolder = snapshotFolder; } void CaptureProcessVideoThread::setCaptureListWidget(QListWidget *captureListWideget){ m_captureListWideget = captureListWideget; } void CaptureProcessVideoThread::run(){ Mat frame, frameToSave; QTextStream cout(stdout); // index for snapshots static int lastSnapIndex = 0; if (m_video->isOpened()){ *m_video >> frame; *m_video >> frameToSave; // try to find chessboard corners // if found draw them int chessBoardFlags = CALIB_CB_ADAPTIVE_THRESH | CALIB_CB_NORMALIZE_IMAGE; vector<Point2f> pointBuf; Settings s; // s.boardSize = Size(19, 11); s = getCalibrationSettings(); bool found = findChessboardCorners( frame, s.boardSize, pointBuf, chessBoardFlags); if (found){ drawChessboardCorners( frame, s.boardSize, Mat(pointBuf), found ); } // create QImage from frame QImage qimg(frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888); QImage qimgToSave(frameToSave.data, frameToSave.cols, frameToSave.rows, frameToSave.step, QImage::Format_RGB888); // update the Graphvis the view with the new frame m_pixmap->setPixmap( QPixmap::fromImage(qimg.rgbSwapped()) ); m_graphicsView->fitInView(m_pixmap, Qt::KeepAspectRatio); // take snapshot if asked for if(m_takeSnapshot){ QString fileName = m_snapshotFolder + QString("/Capture_%1d.png").arg(++lastSnapIndex, 5, 10, QChar('0')); QFileInfo fileInfo(fileName); qimgToSave.save(fileName); // add the snapshot to list of snapshots m_captureListWideget->addItem(new QListWidgetItem( QIcon(fileName), fileInfo.fileName())); m_takeSnapshot = false; } } } void CaptureProcessVideoThread::setGraphicsView(QGraphicsView *graphicsView){ m_graphicsView = graphicsView; } void CaptureProcessVideoThread::captureSnapshot(){ m_takeSnapshot = true; }
-
You should avoid re-creating the pixmap, once you have it with the correct size, just memcpy the content of the processed frame into place.
You should also avoid calling fitInView from that external thread. GUI operation should only happen in the GUI thread. You're lucky it doesn't crash.
Same goes for the snapshot handling.
Did you try to get some performance number from your run method ?
-
so .... you mean sth like: QPixmap::loadFromData
I tried this but no luck so far....
QByteArray arr; QBuffer buffer(&arr); buffer.open(QIODevice::WriteOnly); qimg.save(&buffer); m_pixmap->pixmap().loadFromData(arr);
Looks like I have done correctly before :
https://www.qtcentre.org/threads/56871-QGraphicsPixmapItem-repaint-immediately
No errors ... but I don't see anything...
-
The source code for setPixmap is like below:
void QGraphicsPixmapItem::setPixmap(const QPixmap &pixmap) { Q_D(QGraphicsPixmapItem); prepareGeometryChange(); d->pixmap = pixmap; d->hasShape = false; update(); }
from here: https://code.woboq.org/qt5/qtbase/src/widgets/graphicsview/qgraphicsitem.cpp.html#9691
The only way seems to keep access and update d->pixmap directly using a memcopy sth like that just to avoid changing the location of the image to be viewed in the memory.
am I correct?
QPixmap QGraphicsPixmapItem::pixmap() const { Q_D(const QGraphicsPixmapItem); return d->pixmap; }
-
It's the idea. Avoid re-creating objects if possible. By the way, can't you get OpenCV to directly give you image data in RGB rather than BGR ? That would avoid a copy operation.
-
@mikeitexpert thanks mike, can you share please the: #include <utils/settings.h>
#include <utils/utils.h> ?