Multiple qml video outputs in different windows

felsi

I want to display a video simultaneously in different windows.
I was looking for a solution to this problem, but i found only unsatisfying and outdated answers.
This post was originally meant to be an answer, but i decided to start a new thread.

Solution 1:

This would be the most obvious and efficient solution, but unfortunately this doesn't work in different windows.

    VideoOutput{
        id: video_output
    }
    
    ShaderEffectSource{
        sourceItem: video_output
    }

Solution 2:

So i figured out this solution.
The idea is to have 1 primary QVideoSink and multiple secondary QVideoSinks, which share the same QVideoFrame.
This solution also allows the creation and destruction of video outputs in any order.

The C++ part:

class MediaPlayer : public QMediaPlayer
{
    Q_OBJECT

public slots:
    void setVideoOutput(QObject *video_output);
    void removeVideoOutput(QObject *video_output);
    void videoFrameChanged(const QVideoFrame &frame) const;

private:
    QList<QObject*> video_outputs;
    QPointer<QVideoSink> primary_video_sink = NULL;
    QList<QPointer<QVideoSink>> secondary_video_sinks;

    void updateVideoOutputs();
    void setPrimaryVideoSink(QVideoSink *video_sink);
    void setSecondaryVideoSink(QVideoSink *video_sink);
}

void MediaPlayer::setVideoOutput(QObject *video_output)
{
    if(this->video_outputs.contains(video_output))
        return;

    this->video_outputs.append(video_output);
    connect(video_output, SIGNAL(destroyed(QObject*)), this, SLOT(removeVideoOutput(QObject*)));
    this->updateVideoOutputs();
}

void MediaPlayer::removeVideoOutput(QObject *video_output)
{
    this->video_outputs.removeAll(video_output);
    this->updateVideoOutputs();
}

void MediaPlayer::updateVideoOutputs()
{
    // update video output and primary video sink
    if(this->video_outputs.size()>0 && this->videoOutput()!=this->video_outputs.at(0))
    {
        QMediaPlayer::setVideoOutput(this->video_outputs.at(0));
        this->setPrimaryVideoSink(this->video_outputs.at(0)->property("videoSink").value<QVideoSink*>());
    }

    // update secondary video sinks
    this->secondary_video_sinks.clear();
    for(int i=1;i<this->video_outputs.size();i++)
    {
        this->setSecondaryVideoSink(this->video_outputs.at(i)->property("videoSink").value<QVideoSink*>());
    }
}

void MediaPlayer::setPrimaryVideoSink(QVideoSink *video_sink)
{
    if(this->primary_video_sink==video_sink)
        return;

    if(this->primary_video_sink!=NULL)
        disconnect(this->primary_video_sink, SIGNAL(videoFrameChanged(QVideoFrame)), this, SLOT(videoFrameChanged(QVideoFrame)));

    this->primary_video_sink = video_sink;
    connect(this->primary_video_sink, SIGNAL(videoFrameChanged(QVideoFrame)), this, SLOT(videoFrameChanged(QVideoFrame)));
}

void MediaPlayer::setSecondaryVideoSink(QVideoSink *video_sink)
{
    if(this->secondary_video_sinks.contains(video_sink))
        return;

    this->secondary_video_sinks.append(video_sink);
}

void MediaPlayer::videoFrameChanged(const QVideoFrame &frame) const
{
    for(int i=0;i<this->secondary_video_sinks.size();i++)
    {
        if(this->secondary_video_sinks.at(i)!=NULL)
            this->secondary_video_sinks.at(i)->setVideoFrame(frame);
    }
}

The QML part:

    VideoOutput{
        id: video_output

        //media_player is a context property
        Component.onCompleted: media_player.setVideoOutput(video_output)
    }

Now, i was wondering, because i haven't found anyone else suggesting this solution or even mentioning sinks and frames.
To me, this seems to be an acceptable solution and i hope it will help others.

Maybe someone could answer the following:
Are there any other performance considerations?
What impact on memory and cpu usage are there and why?

felsi

A bit more detailed, I want to run my media player on a laptop with an embedded video and simultaneously i want to show the video in public on a video projector or a video wall. Therefore it is very important, that the videos run smoothly. Unfortunately i encountered jitters, when i do stuff like resizing the media player. After searching and reading about scenegraph, render thread, etc i came up with the following:

I start a dedicated client and send frames to it.
The WindowVideo class contains a QVideoSink and can be used like and aside of other embedded QML video outputs. It consists mainly of two threads. The FrameWriter, which writes frames into a shared memory. And the ProcessHandler, which starts and
communicates with the client program. The videoFrameChanged signal is emitted at a constant rate from the ffmpeg thread and is directly connected to the FrameWriter, where a shallow copy of the frame is created.
The FrameWriter thread waits until a new frame is available. It than copies the frame data into a shared memory segment and sends the update instruction via the ProcessHandler to the client.
The client consists mainly of the FrameReader thread.

Steps per frame:

ffmpeg: creates shallow copy of frame
writer: maps the frame data to ram
writer: creates or reuses shared memory
writer: memcpy frame data from ram to shared memory
handler: sends update instruction to client
client: receives message and invokes reader
reader: creates frame
reader: maps frame to ram
reader: memcpy frame from shared memory to ram
reader: unmaps frame
client: sets the frame at the video sink

And it works better than expected, writing a frame to buffer takes about 1 or 2 ms and reading takes about 3 or 4 ms.
But of course, all the extra steps, which must be taken plus running a whole client program comes at a cost.

So my question is:
Is there another, simpler, more efficient way to achieve this?

SGaist

Hi,

If you have several clients, shouldn't you rather have a dedicated video stream server using for example GStreamer and then your applications connecting to it ?

felsi

@SGaist
Thank you for your answer!
No, for now, i am happy with just one window, "client" is maybe a bit misleading, although a network approach would be nice.
With my solution it should also be possible to start several windows using the same shared memory.
But you made a good point, i was looking for other solutions and i also read parts of the gstreamer docs, but there were to many sticking points for me.

I would have to compile gstreamer for every platform, link it and ship it with my application. I didn't try, but i guess, it would take me longer than i expect...
Using gstreamer as a replacement for the qt multimedia module and do all the media stuff by myself is a lot of work just to avoid a few skipped frames. But it would be great and maybe, some day, some feature will force me to. (blending between tracks, applying filter, etc)
You seem to have a lot of experience, can you estimate the workload? What are the great pitfalls? Would you recommend gstreamer? Or (without rtsp) ffmpeg or libvlc? Gstreamer has a nice documentation and qml example projects...
Using gstreamer by connecting it to the already existing and working qt multimedia module.
I would try to extend the qmlsrc example and take the frame as a source for the gstreamer pipeline. But i am not sure, if this is a good idea at all. And i don't think, that it's simpler or more efficient.
It would have the great advantage, that nothing will break, if my attempts fail.

felsi

Oh, sorry, you wrote dedicated server.
So you suggest to start a standalone video server and replace my QMediaPlayer subclass with a class that controls the server via requests to load, play, etc.? Thank you very much for this new view of point.

My player is a normal player for day to day use on phone, tablet and desktop. But i do fancy stuff like restoring positions, switching active video tracks while playing and make fadeouts when skipping videos. So the way to go for me, would be to integrate gstreamer and control the pipeline directly via c++. Or not?

Using a dedicated video server sounds more suitable for a complex stage setup. I just want to plug in a projector and drag my video window onto. I use the video window also as a fullscreen mode on a single display. I was more looking for some magic video widget, which bypasses the gui thread and renders a video at a guaranteed framerate. e.g. i drew frames directly on a raster window, with very, very poor results...

The shared memory solution has a low latency and a high framerate, but it increases the cpu usage about 30%...

SGaist

If you have the required hardware, you might want to try QtQuick. It should use hardware acceleration and the rendering might be faster.

felsi

@SGaist
My problem is not rendering related. The problem is, that the gui thread blocks because of my rather complex qml scene (js, bindings, etc). And as the video is part of the scene, it can't be updated faster than the rest of the scene.
While this is no big deal for most players (small ui, embedded video), it is pretty noticeable in mine (complex ui, windowed video).

Drawing the frames with a separate thread directly on a raster window was more an ugly experiment.
Using QtQuick would either mean using the existing gui thread, which blocks, or using a second, which is not possible.
But i was hoping, that maybe someone has optimized such an approach of bypassing the gui thread.
This would be especially useful, if the rest of the scene could be buffered and layed on top.
Every player would benefit from a guaranteed smooth running video.
It's definitely beyond my skills and resources...

I'll just live with my weird and inefficient 2 processes workaround or implement a rtsp solution.
I would also appreciate tips about doing this without copying every frame 5 times...

Thanks for your replies!

Liyong Zhou

(1) Recommend you to profile the running with various tools. Like system profiling, and QT
https://doc.qt.io/qtcreator/creator-cache-profiler.html
Notice the profiling depends on systems, like windows, linux, android, etc, since their resource profiles are quite different.
(2) Use gstreamer as a guidance. Linux like Ubuntu has gstreamer preinstalled. If using ffmpeg, you can quickly set up a gstreamer pipeline to do all video things without the GUI or QT. And compare performance with your GUI version.
(3) as suggested by SGaist, trying to use hardware acceleration like GPU, and openGL memory. Gstreamer plugins have openGL memory support to avoid extra memory copy. Although you may not use them in the final product, use this as a guidance to compare to existing solution.
(4) as suggested, you can run gstreamer result to a QT window so you don't have to do all the memory and thread management. There are examples on the web, and the basic flow is: QT QML "window { id:win1 ... }", and C++ get window ID, then use "gst_video_overlay_set_window_handle (sink, wID)".
(5) As to 2nd display, you can set up another "window { id:win2 ...}" with different x,y positions as window extension, and get another window ID, then send gstreamer output to that wID2.

Ronel_qtmaster

@felsi

Do you mean something like this? It is a software i am working on