QMediaPlayer unable to seek accurately on MacOS

donquibeats

Hello

I've got a QVideoWidget, QMediaPlayer test app running, having solved my media path issue from a couple of days ago.

Now I notice that running the same code, on the same video file, is giving different behaviours on Windows and Mac.

The app needs to be able to frame-step through the video, one frame at a time, so I have 'forward' and 'back' buttons which call the QMediaPlayer's setPosition() method and give it a value which is 40ms lower or higher than the existing position value. (My test files are 25 frames per second so 40ms = 1 frame).

On Windows, this works great. It frame-steps perfectly, one frame at a time, without skipping or doubling any. Lovely.

However on Mac, the QMediaPlayer jumps to a frame that is sometimes two or three seconds before the correct position. It behaves as though it is going to the nearest keyframe prior to that time, or something like that. Small changes of 40ms (1 frame) to the position do not work- only large changes, sometimes over 2000 or 3000ms, will make the video position change.

I can see from this page that AV Foundation is used when on a Mac. According to this AV Foundation documentation, the 'seek' method has optional extra parameters called 'toleranceBefore' and 'toleranceAfter' which allow you to request sample-accurate seeking behaviour. If you don't set those optional parameters, the documentation acknowledges that "the actual time to which the player moves may differ from the time you requested"- which isn't good enough for my app's needs.

However, it looks like the tolerance parameters are not exposed in the QMediaPlayer method- so as far as I can see, I have no way in my PyQt5 code of insisting on zero tolerance in QMediaPlayer.setPosition()? Is that the case?

Is there a way I can customise my setPosition() code or add some Mac-specific code which would allow me to seek precisely using AV Foundation, without breaking the Windows, Media Foundation-based behaviour?

SGaist

Hi,

One possible way would be to rebuild the backend with the tolerance you need.

donquibeats

@SGaist said in QMediaPlayer unable to seek accurately on MacOS:

One possible way would be to rebuild the backend with the tolerance you need.

Thanks for the suggestion. Please could you be a bit more specific about what you mean by "rebuild the backend"?

I'm new to Qt and PtQt5, and fairly new to Python, so I was hoping there was a solution that wouldn't involve an immediate dive into the deep end of Qt.

ziggx

@donquibeats

The problem is with your source video - you need to have all i-frames when working with QMediaPlayer on mac. Most videos only have i-frames every second or so. You can convert your video using ffmpeg:

ffmpeg -i VIDEO-TO-CONVERT -preset medium -codec:v libx264 -intra -pix_fmt yuv420p OUTPUT-VIDEO

Hope this helps
Ziggx

donquibeats

Thanks @ziggx for the suggestion.

In this particular set-up, with a lot of video files and quite fast turnaround required, converting the video files is a potential workaround rather than a perfect solution, as it adds extra overhead. However it might be something that I can append to our workflow purely for Mac users (who are in a minority here), so I'll give it some thought.

donquibeats

Just to add to this, unfortunately @ziggx's suggestion of re-encoding with FFmpeg to get every frame to be an iframe doesn't seem to work either. The output video file, which is 2.5 times the size of the original, reports as having keyframes on every frame according to FFprobe- but still, on a Mac, QMediaPlayer will still not seek precisely.

Testing with the setPosition() and getPosition() methods, it would appear that the QMediaPlayer still wants to round down to the nearest 1000, so at 25 frames per second, trying to go to frame 74 (setPosition(2960)) will actually go to frame 50, and getPosition() immediately reports a value of 2000.

So as well as the keyframes issue, is it possible that in Mac-specific QMediaPlayer implementation, there's some inherent position rounding going on?

I also tried alternative FFmpeg parameters -c:v libx264 -x264opts keyint=1 and that gave the same result, I'm thinking that's because -intra is a synonym for keyint=1 but I haven't checked this.

SGaist

@donquibeats said in QMediaPlayer unable to seek accurately on MacOS:

@SGaist said in QMediaPlayer unable to seek accurately on MacOS:

One possible way would be to rebuild the backend with the tolerance you need.

Thanks for the suggestion. Please could you be a bit more specific about what you mean by "rebuild the backend"?

I'm new to Qt and PtQt5, and fairly new to Python, so I was hoping there was a solution that wouldn't involve an immediate dive into the deep end of Qt.

Sorry but yes I am suggesting to re-build the AV foundation QtMultimedia backend, but only that, no need to rebuild the whole of Qt.

donquibeats

@SGaist said in QMediaPlayer unable to seek accurately on MacOS:

Sorry but yes I am suggesting to re-build the AV foundation QtMultimedia backend, but only that, no need to rebuild the whole of Qt.

Thanks for clarifying.

After project management discussions we've decided that frame seeking on a Mac is definitely required but not immediately, so we've got this adjustment on our roadmap as something to come back to in a month or two's time. So I'll come back to that thread at that time.

I would hope that adding extra optional parameters to the QMediaPlayer setPosition() functionality would be a benefit to other users, not just our project, so hopefully we'll be able to feed something back as a consequence.

SGaist

That would be very nice ! Thanks

J0Nes

@donquibeats

Hey, we are facing the very same issue(s) on macOS. Have you been able to come back to it already and could share your experience?
Thanks!

J0Nes

Hey again
I could fix the issue by rebuilding the backend as suggested by @SGaist, see the patch below.

diff --git a/src/plugins/avfoundation/mediaplayer/avfmediaplayersession.mm b/src/plugins/avfoundation/mediaplayer/avfmediaplayersession.mm
index 2309221c..444b08ef 100644
--- a/src/plugins/avfoundation/mediaplayer/avfmediaplayersession.mm
+++ b/src/plugins/avfoundation/mediaplayer/avfmediaplayersession.mm
@@ -684,9 +684,9 @@ void AVFMediaPlayerSession::setPosition(qint64 pos)
     if (duration() > 0)
         pos = qMin(pos, duration());
 
-    CMTime newTime = [playerItem currentTime];
-    newTime.value = (pos / 1000.0f) * newTime.timescale;
-    [playerItem seekToTime:newTime];
+    int32_t timeScale = playerItem.asset.duration.timescale;
+    CMTime newTime = CMTimeMakeWithSeconds(pos / 1000.0, timeScale);
+    [playerItem seekToTime:newTime toleranceBefore:kCMTimeZero toleranceAfter:kCMTimeZero];
 
     Q_EMIT positionChanged(pos);

edit: I also found that the computation of newTime is sometimes incorrect due to a fishy looking timeScale. I figured that the code change above improves seeking significantly when the video is still in m_state PausedState.

VaL Doroshchuk

https://bugreports.qt.io/browse/QTBUG-81804