Unsolved Modify frames from Camera/Video before displaying
-
Hello,
I am developing QMediaPlayer application for Windows platform using Qt5.12.I want to perform some OpenCV operations on video frames before displaying it. I do expect there would be some delay due to expensive OpenCV operations. However, video and audio should be synced.
Also, I want to have a framework where if I am applying any Opencv operations, the video should play without any delay.
I see lot of options ->
- QT with LibVLC
- QAbstractVideoSurface(tried seems slow to convert from QVideoFrame to QImg and Opencv::Mat)
- QAbstractVideoFilter
- QtAV lib
However, I am not sure which approach would be best with minimal delay and video audio sync.
-
Hi,
If you want minimal perturbation, you have to go low level and do your preprocessing in advance thus pre-loading frames to have enough time to apply your operations. You'll have to benchmark things. You can't expect something nice if it requires 100ms per frame for a movie at 24fps.
What kind of operations do you have in mind ?
-
@SGaist ,
As of now, I plan to apply- Kalman filter
- Color detection
- Smoothing, etc.
In Opencv, these algorithms do work at 24 fps. The major delay is converting from QVideoFrame to OpenCV Mat.
I want to avoid this conversion time
-
What about doing it the other way around, do everything with OpenCV and only the display at the end with Qt ?
-
@SGaist ,
I am have Opencv based VideoPlayer using both Thead and Timer based approach. However, I face following issues:- How much delay should I give. If exact delay is given as below:
45 secs video would finish in 55 secs to be approx without any operations. I don't understand how can I minimize this delay.
int delay = (1000/frameRate); while(!stop){ if (!capture.read(frame)) { stop = true; } if (frame.channels()== 3){ cv::cvtColor(frame, RGBframe, CV_BGR2RGB); img = QImage((const unsigned char*)(RGBframe.data), RGBframe.cols,RGBframe.rows,QImage::Format_RGB888); } else { img = QImage((const unsigned char*)(frame.data), frame.cols,frame.rows,QImage::Format_Indexed8); } emit processedImage(img); QThread::msleep(delay); }
- Sync with audio. I see some solutions using opencv SDL. However, I am currently overwhelmed with situations and not exactly sure which way to go.
- How much delay should I give. If exact delay is given as below:
-
How are you reading the audio data with OpenCV ?
I just realize something, IIRC, there are frame format that allows construction without copy of the data. Did you check that ? -
@SGaist said in Modify frames from Camera/Video before displaying:
IIRC
My mistake. ffmpeg is used to read audio and OpenCV to read video.
there are frame format that allows construction without copy of the data
- I am not sure which frame format you are saying and how it would help resolve the issue.
-
It looks like ffmpeg is directly supported by OpenCV. Did you check that ?
As for the format, if using something like RGB 8 bit, you will have data that are following the same memory layout be it for OpenCV Mat or QImage so no conversation needed.
-
I saw that post and there are few tutorials for audio video sync with opencv.
However, for opencv Mat to QImg, I need to call BGR to RGB converter. There is still some delay.
-
Are you using OpenCV to do that conversion ?
-
@SGaist
I have tried both OpenCV convert and rgbswapped of QT. There is not much difference in performance -
@magicstar this is example with openCV to modify frames:
cv::VideoCapture CaptureVideo; bool val = CaptureVideo.open(videoFile.toStdString()); if(CaptureVideo.isOpened()) { while(1) { if(isFinished || isStopped) { CaptureVideo.release(); break; } cv::Mat frame; CaptureVideo >> frame; if(!frame.empty()) { QImage qimg(frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888); // Editing frame with date and time or color change etc.. { // qimg.fill(Qt::red); QPainter p; if (!p.begin(&qimg)) qDebug()<<"Not"; p.setPen(QPen(Qt::yellow)); // p.setOpacity(0.5); p.setFont(QFont("Times", 18, QFont::Bold)); p.drawText(qimg.rect(), Qt::AlignVertical_Mask, " "+QDateTime::currentDateTime().toString("ddd MMMM dd yyyy HH:mm:ss")); p.end(); } } } }