QVideoFrame does not deliver correct startTime() and endTime() / QMediaPlayer issue

Amazonasmann

Hi,
I want to process videos frame by frame. For doing this I found this solution that uses an implementation of QAbstractVideoSurface. Which works fine when I use only play/pause/stop of QMediaPlayer. I require the use of QVideoFrame::startTime() and QVideoFrame::endTime() to synchronize the video to some other data. After I have connected a QSlider to the QMediaplayer, I found out that after usage of
QMediaPlayer::setPosition() function the QVideoFrame::startTime() does not deliver correct results anymore (The QVideoFrame::startTime() always delivers the same value while slider is being dragged and the frame clearly changes). This alone is a problem but could be solved if the QAbstractVideoSurface::present() function would be synchronized to the QMediaPlayer, but it is not. Counting the frames inside my QAbstractVideoSurface implementation was also no success.

What I need is a reliable way to get the current frame number inside the QAbstractVideoSurface::present() function. With this I could recalculate the correct startTime() and endTime(). Does anyone have suggestions?

Basically I tried this, which still does not work for dragging sliders:


bool VideoFrameGrabber::start(const QVideoSurfaceFormat& format)
{
   CurrentFrameNumber = 0;
}

void VideoFrameGrabber::stop()
{
   CurrentFrameNumber = 0;
}

bool VideoFrameGrabber::present(const QVideoFrame& frame)
{
   //CurrentFrameNumber is out of sync with the actually displayed frame when slider is being dragged
   CurrentFrameNumber++;
   //Here frame.startTime() delivers wrong values if the video was moved by QMediaPlayer::setPosition()
}

void VideoFrameGrabber::Slot_MediaPlayerSeeked(int milliseconds)
{
//This is connected to
//  connect(ui->slider_Video, &QSlider::sliderMoved, ...);
   CurrentFrameNumber = CalculateFrameNumber();
}

qint64 VideoFrameGrabber::CalculateFrameNumber()
{
   int fps = ThePlayer->metaData("VideoFrameRate").toInt();
   qreal hertz = 1000 / fps;
   qreal position = ThePlayer->position();
   qint64 frameNumber = position / hertz;
   return frameNumber;
}

Additional Info: After refining my search terms, I found this question from 6 years ago. This is exactly my problem. I would like to "block" QMediaPlayer in a way that there are no skipped frames.

SGaist

Hi,

Out of curiosity, why is herz 1000/fps ?

That looks strange.

Kent-Dorfman

This post is deleted!

Amazonasmann

Hi @SGaist ,
I do this to calculate the time how long each frame should be shown. For example I have a video with 25 FPS. I know a second has 1000ms...so I divide 1000 / 25 and get 40 as a result. This means each frame has a length of 40ms..By knowing the time I of the QMediaPlayer, let's say 2000ms I should be in the 2000/40 (50) frame. This calculation happens to be incorrect for QMediaPlayer but this is more due to some internal reasons. If I use OpenCV and have more control over video frames this is one of the ways to calculate the current frame. I know the name herz is not the best for this, please ignore the variable name :-).

Back to the topic:
By now it most certainly looks like a Qt bug for me. Each frame has no metadata() available, which they should have. Having the meta data from each frame would solve my problem. I tested this with multiple files, recorded from different applications (.mp4 files) and each time there is no metadata() available in a QVideoFrame.

Also the startTime() and endTime() (The presentation timestamp which SHOULD be coming from the metadata) is only present when the video is started from the beginning and is lost once you set the time in the QMediaPlayer manually. Each frame should have meta data available since it is available in the media file (I double checked this via ffmpeg).

SGaist

What OS are you on ?

Amazonasmann

I am using Windows Qt 5.13.0 MSVC2017 64bit

SGaist

How are you making sure that these meta data are available ?
Does the native windows media player show them ?

Note that depending on how you generate your video, you won't have a full frame per image.

Amazonasmann

The map for meta data in any QVideoFrame is never filled. When I use a command line tool like ffmpeg to write down all meta data for each frame, it works just as you expect. There is definitely meta data available in the videos. I want the meta data for each frame not the whole file (framerate etc. is available in the meta data of the video file).

The weird thing is, why I think it could be a bug: When I play the video from start to end, just by
using QMediaPlayer::Play() and QMediaPlayer::Pause() and QMediaPlayer::stop() the QVideoFrame::startTime() and QVideoFrame::endTime() always delivers correct values.If I jump into the video with QMediaPlayer::setPosition, it is no longer correct until I do QMediaPlayer::Stop().

Another thing is, if you use QVideoProbe instead of QAbstractVideoSurface, there is never a correct startTime and endTime for any frame.

If you have some time available you can reproduce it by using the standard example "media player" inside Qt Creator's welcome screen and change the following:

in histogramwidget.cpp

#include <QDebug>
void HistogramWidget::processFrame(QVideoFrame frame)
{
 qInfo() <<" START:" <<frame.startTime() << " END: "<< frame.endTime(); // this is new
.
. //rest of the code
.
}

This will always output "START -1 END -1" in my case. If you set the video output of the mediaplayer to a QAbstractVideoSurface, the QAbstractVideoSurface::present() function will get you a correct startTime() and endTime()...but only if you never jump in the video.

Amazonasmann

I opened up a bug report and as it seems, this was removed some time ago. https://bugreports.qt.io/browse/QTBUG-77849
So my solution can be found in this link.

Basically I need to use a different backend for my QMediaplayer (Instead of directshow I use WMF now) which gives me a proper startTime() and endTime() for each frame.

I am still puzzled on how it is not possible to get the current frame as a number with QMediaPlayer.
Let's say in QMediaPlayer you do play() -> pause() -> play() -> pause() you will always advance frames but the time will look like this: 100ms->101ms->102ms->103ms...so computing QMediaPlayer::position() / framerate is not the correct way to get the actual current frame number. Which is important for image processing.

Kent-Dorfman

@amazonasmann
I would guess that getting the frame number from media player isn't a proiority...because for a given timestamp the frame number is not constant. codecs have the ability to drop frames if they get behind in decoding. Also, for progressive frame codecs, you CANT start on a P frame. The only valid start frame must be I frames.

Amazonasmann

@Kent-Dorfman True. I agree with that. But what if you have a third party software that outputs data frame by frame and you want to visualize it in Qt? Frameworks like OpenCV (a computer vision library) mostly use a simple get() function which returns the next frame and they simply output this number besides additional information which was processed.
In my case I have both: Data which has a presentation timestamp for each frame and data which uses simple frame number as synchronization. Now I have to visualize both informations in the same frame.
The only way I can think of right now is using the QVideoFrame::startTime() and QVideoFrame::endTime() function...to calculate the frame number which frame I am currently processing.

As I am a Qt fan, I always start a new project which uses Qt and try to not use any more libraries than really necessary, because I could have used OpenCV to solve this. However as we all know, Qt has some huge advantages. But nevertheless I am still a bit disappointed that you can not really process a video frame by frame easily.

Kent-Dorfman

@amazonasmann said in QVideoFrame does not deliver correct startTime() and endTime() / QMediaPlayer issue:

Now I have to visualize both informations in the same frame.

Well I guess you had better throw out the idea of using the Qt multimedia system and write your own decoder and displayer. I've done it in Qt4 where we had to support custom codecs delivering video over wireless links. decode your frames manually into a suitable pixel format buffer and then use an x11 video extension to map that pixel data into an application window.