QVideoFrame does not deliver correct startTime() and endTime() / QMediaPlayer issue

Kent-Dorfman

This post is deleted!

Amazonasmann

Hi @SGaist ,
I do this to calculate the time how long each frame should be shown. For example I have a video with 25 FPS. I know a second has 1000ms...so I divide 1000 / 25 and get 40 as a result. This means each frame has a length of 40ms..By knowing the time I of the QMediaPlayer, let's say 2000ms I should be in the 2000/40 (50) frame. This calculation happens to be incorrect for QMediaPlayer but this is more due to some internal reasons. If I use OpenCV and have more control over video frames this is one of the ways to calculate the current frame. I know the name herz is not the best for this, please ignore the variable name :-).

Back to the topic:
By now it most certainly looks like a Qt bug for me. Each frame has no metadata() available, which they should have. Having the meta data from each frame would solve my problem. I tested this with multiple files, recorded from different applications (.mp4 files) and each time there is no metadata() available in a QVideoFrame.

Also the startTime() and endTime() (The presentation timestamp which SHOULD be coming from the metadata) is only present when the video is started from the beginning and is lost once you set the time in the QMediaPlayer manually. Each frame should have meta data available since it is available in the media file (I double checked this via ffmpeg).

SGaist

What OS are you on ?

Amazonasmann

I am using Windows Qt 5.13.0 MSVC2017 64bit

SGaist

How are you making sure that these meta data are available ?
Does the native windows media player show them ?

Note that depending on how you generate your video, you won't have a full frame per image.

Amazonasmann

The map for meta data in any QVideoFrame is never filled. When I use a command line tool like ffmpeg to write down all meta data for each frame, it works just as you expect. There is definitely meta data available in the videos. I want the meta data for each frame not the whole file (framerate etc. is available in the meta data of the video file).

The weird thing is, why I think it could be a bug: When I play the video from start to end, just by
using QMediaPlayer::Play() and QMediaPlayer::Pause() and QMediaPlayer::stop() the QVideoFrame::startTime() and QVideoFrame::endTime() always delivers correct values.If I jump into the video with QMediaPlayer::setPosition, it is no longer correct until I do QMediaPlayer::Stop().

Another thing is, if you use QVideoProbe instead of QAbstractVideoSurface, there is never a correct startTime and endTime for any frame.

If you have some time available you can reproduce it by using the standard example "media player" inside Qt Creator's welcome screen and change the following:

in histogramwidget.cpp

#include <QDebug>
void HistogramWidget::processFrame(QVideoFrame frame)
{
 qInfo() <<" START:" <<frame.startTime() << " END: "<< frame.endTime(); // this is new
.
. //rest of the code
.
}

This will always output "START -1 END -1" in my case. If you set the video output of the mediaplayer to a QAbstractVideoSurface, the QAbstractVideoSurface::present() function will get you a correct startTime() and endTime()...but only if you never jump in the video.

Amazonasmann

I opened up a bug report and as it seems, this was removed some time ago. https://bugreports.qt.io/browse/QTBUG-77849
So my solution can be found in this link.

Basically I need to use a different backend for my QMediaplayer (Instead of directshow I use WMF now) which gives me a proper startTime() and endTime() for each frame.

I am still puzzled on how it is not possible to get the current frame as a number with QMediaPlayer.
Let's say in QMediaPlayer you do play() -> pause() -> play() -> pause() you will always advance frames but the time will look like this: 100ms->101ms->102ms->103ms...so computing QMediaPlayer::position() / framerate is not the correct way to get the actual current frame number. Which is important for image processing.

Kent-Dorfman

@amazonasmann
I would guess that getting the frame number from media player isn't a proiority...because for a given timestamp the frame number is not constant. codecs have the ability to drop frames if they get behind in decoding. Also, for progressive frame codecs, you CANT start on a P frame. The only valid start frame must be I frames.

Amazonasmann

@Kent-Dorfman True. I agree with that. But what if you have a third party software that outputs data frame by frame and you want to visualize it in Qt? Frameworks like OpenCV (a computer vision library) mostly use a simple get() function which returns the next frame and they simply output this number besides additional information which was processed.
In my case I have both: Data which has a presentation timestamp for each frame and data which uses simple frame number as synchronization. Now I have to visualize both informations in the same frame.
The only way I can think of right now is using the QVideoFrame::startTime() and QVideoFrame::endTime() function...to calculate the frame number which frame I am currently processing.

As I am a Qt fan, I always start a new project which uses Qt and try to not use any more libraries than really necessary, because I could have used OpenCV to solve this. However as we all know, Qt has some huge advantages. But nevertheless I am still a bit disappointed that you can not really process a video frame by frame easily.

Kent-Dorfman

@amazonasmann said in QVideoFrame does not deliver correct startTime() and endTime() / QMediaPlayer issue:

Now I have to visualize both informations in the same frame.

Well I guess you had better throw out the idea of using the Qt multimedia system and write your own decoder and displayer. I've done it in Qt4 where we had to support custom codecs delivering video over wireless links. decode your frames manually into a suitable pixel format buffer and then use an x11 video extension to map that pixel data into an application window.