Modify frames from Camera/Video before displaying

[magicstar]

Hello,
I am developing QMediaPlayer application for Windows platform using Qt5.12.

I want to perform some OpenCV operations on video frames before displaying it. I do expect there would be some delay due to expensive OpenCV operations. However, video and audio should be synced.

Also, I want to have a framework where if I am applying any Opencv operations, the video should play without any delay.

I see lot of options ->

1. QT with LibVLC
2. QAbstractVideoSurface(tried seems slow to convert from QVideoFrame to QImg and Opencv::Mat)
3. QAbstractVideoFilter
4. QtAV lib

However, I am not sure which approach would be best with minimal delay and video audio sync.

[SGaist]

Hi,

If you want minimal perturbation, you have to go low level and do your preprocessing in advance thus pre-loading frames to have enough time to apply your operations. You'll have to benchmark things. You can't expect something nice if it requires 100ms per frame for a movie at 24fps.

What kind of operations do you have in mind ?

[magicstar]

@SGaist ,
As of now, I plan to apply

1. Kalman filter
2. Color detection
3. Smoothing, etc.

In Opencv, these algorithms do work at 24 fps. The major delay is converting from QVideoFrame to OpenCV Mat.

I want to avoid this conversion time

[SGaist]

What about doing it the other way around, do everything with OpenCV and only the display at the end with Qt ?

[magicstar]

@SGaist ,
I am have Opencv based VideoPlayer using both Thead and Timer based approach. However, I face following issues:

1. How much delay should I give. If exact delay is given as below:
   45 secs video would finish in 55 secs to be approx without any operations. I don't understand how can I minimize this delay.

int delay = (1000/frameRate);
    while(!stop){
        if (!capture.read(frame))
        {
            stop = true;
        }
        if (frame.channels()== 3){
            cv::cvtColor(frame, RGBframe, CV_BGR2RGB);
            img = QImage((const unsigned char*)(RGBframe.data),
                              RGBframe.cols,RGBframe.rows,QImage::Format_RGB888);
        }
        else
        {
            img = QImage((const unsigned char*)(frame.data),
                                 frame.cols,frame.rows,QImage::Format_Indexed8);
        }
        emit processedImage(img);
        QThread::msleep(delay);
    }

2. Sync with audio. I see some solutions using opencv SDL. However, I am currently overwhelmed with situations and not exactly sure which way to go.

[SGaist]

How are you reading the audio data with OpenCV ?
I just realize something, IIRC, there are frame format that allows construction without copy of the data. Did you check that ?

[magicstar]

@SGaist said in Modify frames from Camera/Video before displaying:

IIRC

My mistake. ffmpeg is used to read audio and OpenCV to read video.

there are frame format that allows construction without copy of the data

• I am not sure which frame format you are saying and how it would help resolve the issue.

[SGaist]

It looks like ffmpeg is directly supported by OpenCV. Did you check that ?

As for the format, if using something like RGB 8 bit, you will have data that are following the same memory layout be it for OpenCV Mat or QImage so no conversation needed.

[magicstar]

@SGaist

I saw that post and there are few tutorials for audio video sync with opencv.

However, for opencv Mat to QImg, I need to call BGR to RGB converter. There is still some delay.

[SGaist]

Are you using OpenCV to do that conversion ?

[magicstar]

@SGaist
I have tried both OpenCV convert and rgbswapped of QT. There is not much difference in performance

[anil_arise]

@magicstar this is example with openCV to modify frames:

cv::VideoCapture CaptureVideo;
bool val = CaptureVideo.open(videoFile.toStdString());
if(CaptureVideo.isOpened())
{
    while(1)
    {
        if(isFinished || isStopped)
        {
            CaptureVideo.release();
            break;
        }
        cv::Mat frame;
        CaptureVideo >> frame;
        if(!frame.empty())
        {
            QImage qimg(frame.data,
                        frame.cols,
                        frame.rows,
                        frame.step,
                        QImage::Format_RGB888);
            // Editing frame with date and time or color change etc.. 
            {
                // qimg.fill(Qt::red);
                QPainter p;
                if (!p.begin(&qimg))
                    qDebug()<<"Not";
                p.setPen(QPen(Qt::yellow));
                // p.setOpacity(0.5);
                p.setFont(QFont("Times", 18, QFont::Bold));
                p.drawText(qimg.rect(), Qt::AlignVertical_Mask, "  "+QDateTime::currentDateTime().toString("ddd MMMM dd yyyy  HH:mm:ss"));
                p.end();
            }
        }
    }
}