Qt Widgets + GStreamer + Overlay + Screen Capture

[SeeRich]

This is really two questions but the code is only slightly different and I didn't want to post it twice.

• How to get text overlay on top of GStreamer video stream.
• How to fix screen capture error imageFromWinHBITMAP_GetDiBits: GetDIBits() failed to get data. (The operation completed successfully.) That occurs after a few successful screen captures.

I will post all the code and the image but first if you would like to build this example, you will need to install GStreamer from here. I am currently developing on Windows, if you are as well, don't forget to add GSTREAMER_ROOT_X86_64/bin to your path as detailed in the link.

The overlay problem:

• I would like this overlay centered in the widget (I can achieve this by listening to resize events and manually positioning if needed). I was just hoping there was a better way.
• Most importantly, I want to get rid of that surrounding black box. No matter what I changed, that was always there.

The code:

• It's a lot but it is the smallest standalone example I could come up with.

#include <QApplication>
#include <QCoreApplication>
#include <QFileDialog>
#include <QLayout>
#include <QMainWindow>
#include <QMenu>
#include <QPainter>
#include <QScreen>
#include <QStandardPaths>
#include <QtGlobal>

extern "C" {
#include <glib-2.0/glib.h>
#include <gst/gst.h>
#include <gst/video/videooverlay.h>
}

#ifdef _WIN32
    #include <windows.h>
#endif

#include <filesystem>
#include <format>
#include <iostream>
#include <string>

using WindowId = void*;

// DEBUG pipeline:
constexpr char debugPipelineDescription[] =
    R"(videotestsrc is-live=true ! videoconvert ! video/x-raw,format=I420,width=720, height=600 ! autovideosink sync=false)";

namespace gstreamer {

GstElement* parse(const std::string& pipelineDescription)
{
    GstElement* result{nullptr};
    GError* error{nullptr};
    result = gst_parse_launch(pipelineDescription.c_str(), &error);
    if(error) {
        std::string msg = std::string("Parsing pipeline failed. Reason: ") + error->message;
        throw std::runtime_error(msg);
    }
    return result;
}

void setState(GstElement* element, GstState state)
{
    GstStateChangeReturn ret = gst_element_set_state(element, state);
    if(GST_STATE_CHANGE_FAILURE == ret) {
        std::string msg =
            std::string("Failed to change the pipeline state to: ") + std::to_string(state);
        throw std::runtime_error(msg);
    }
}

}  // namespace gstreamer

/// Video widget that gstreamer renders to
class GstVideoBacking : public QWidget
{
public:
    GstVideoBacking(QWidget* parent) : QWidget(parent) {}

    virtual ~GstVideoBacking() = default;
};

// Widget responsible for overlays on top of the video stream
class OverlayWidget : public QWidget
{
    Q_OBJECT
public:
    OverlayWidget(QWidget* parent) : QWidget(parent)
    {
        setAttribute(Qt::WA_TranslucentBackground);
        setWindowFlags(Qt::FramelessWindowHint | Qt::WindowStaysOnTopHint);
    }

    virtual ~OverlayWidget() = default;

    virtual void paintEvent(QPaintEvent*) override
    {
        QPainter painter(this);
        painter.setBackground(Qt::transparent);
        painter.setPen(QPen(Qt::yellow, 3));
        painter.setBrush(Qt::NoBrush);

        QString text = tr("OVERLAY TEXT HERE");
        QFont font;
        font.setPointSize(20);
        painter.setFont(font);
        QFontMetrics metrics(font);
        resize(metrics.size(0, text));

        QTextOption options;
        options.setWrapMode(QTextOption::WrapAtWordBoundaryOrAnywhere);
        options.setAlignment(Qt::AlignHCenter | Qt::AlignVCenter);

        painter.drawText(rect(), text, options);
    }
};

class MainWindow : public QMainWindow
{
    Q_OBJECT
public:
    MainWindow()
    {
        resize(720, 600);
        setAttribute(Qt::WA_StyledBackground);
        setStyleSheet("MainWindow { background-color: black; }");

        // We provide a custom context menu for the screenshot
        setContextMenuPolicy(Qt::CustomContextMenu);
        connect(this, &QWidget::customContextMenuRequested, this, &MainWindow::showContextMenu);

        // Force the widget to use a native window.
        setAttribute(Qt::WA_NativeWindow, true);

        // Create video widget that gstreamer will draw on
        _video_widget = new GstVideoBacking(this);
        _video_widget->show();
        setCentralWidget(_video_widget);

        _overlay_widget = new OverlayWidget(this);
        _overlay_widget->raise();

        // Get the video widget window ID for gstreamer
        _window_id = reinterpret_cast<void*>(_video_widget->winId());

        startPipeline();
    }

    ~MainWindow()
    {
        stopPipeline();
        if(_pipeline)
            g_object_unref(_pipeline);
    }

    void showContextMenu(const QPoint& point)
    {
        QMenu contextMenu(tr("Context menu"), this);
        QAction actionCapture(tr("Take Screenshot"), this);
        connect(&actionCapture, &QAction::triggered, this, [this]() { takeScreenshot(); });
        contextMenu.addAction(&actionCapture);
        contextMenu.exec(mapToGlobal(point));
    }

private:
    void startPipeline()
    {
        // Try to parse the pipeline catching any errors
        try {
            qDebug() << "Starting GStreamer pipeline";
            _pipeline = gstreamer::parse(debugPipelineDescription);

            // Listen to synchronous messages so that we can overlay the xwindow later.
            GstBus* bus = gst_pipeline_get_bus(GST_PIPELINE(_pipeline));
            if(bus == nullptr)
                throw std::runtime_error("Could not get pipeline bus...");
            gst_bus_enable_sync_message_emission(bus);
            g_signal_connect(bus, "sync-message", G_CALLBACK(&MainWindow::onSyncMessage), this);
            // Unref bus
            g_object_unref(bus);
            // Set the state to playing.
            gstreamer::setState(_pipeline, GST_STATE_PLAYING);
        } catch(...) {
            qDebug() << "Error: failed to start pipeline";
        }
    }

    void stopPipeline()
    {
        if(!_pipeline)
            return;
        try {
            // No longer listen to synchronous messages.
            GstBus* bus = gst_pipeline_get_bus(GST_PIPELINE(_pipeline));
            if(bus == nullptr)
                throw std::runtime_error("Could not get pipeline bus...");
            gst_bus_disable_sync_message_emission(bus);
            g_signal_handlers_disconnect_by_func(
                bus, (void*)G_CALLBACK(&MainWindow::onSyncMessage), this);
            // Set the state to stopped
            qDebug() << "DEBUG: Shutting down GStreamer pipeline";
            gstreamer::setState(_pipeline, GST_STATE_NULL);
            // Unref bus
            g_object_unref(bus);
            // Reset the pipeline
            _pipeline = nullptr;
        } catch(...) {
            qDebug() << "Error: failed to stop pipeline";
        }
    }

    /// Static callback passed to GStreamer's "C" interface
    static void onSyncMessage(GstBus* inBus, GstMessage* inMessage, gpointer inData)
    {
        auto obj = static_cast<MainWindow*>(inData);
        obj->on_sync_message(inBus, inMessage);
    }

    void on_sync_message(GstBus*, GstMessage* message)
    {
        // Pass the window handle to gstreamer.
        if(gst_is_video_overlay_prepare_window_handle_message(message)) {
            qDebug() << "DEBUG: window handle message received ["
                     << gst_message_type_get_name(GST_MESSAGE_TYPE(message)) << "]";
            gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(GST_MESSAGE_SRC(message)),
                                                (guintptr)_window_id);
        }
    }

    void takeScreenshot() const
    {
        // Determine the screen area that we want to capture
        auto globalXY = _video_widget->mapToGlobal(QPoint(0, 0));
        auto size = _video_widget->size();
        auto screen = qApp->primaryScreen();
        auto capturePixmap =
            screen->grabWindow(0, globalXY.x(), globalXY.y(), size.width(), size.height());
        // Ask the user where they want to save the file.
        auto picturesFolder = QStandardPaths::writableLocation(QStandardPaths::PicturesLocation);
        auto path = std::filesystem::path(picturesFolder.toStdString()) / "VideoCapture.png";
        auto defaultPath = QString::fromStdString(path.string());
        QString fileName = QFileDialog::getSaveFileName(
            _video_widget, tr("Save File"), defaultPath, tr("Images (*.png)"));
        // Save the screen capture
        capturePixmap.save(fileName);
    }

private:
    GstElement* _pipeline{nullptr};
    WindowId _window_id{nullptr};
    GstVideoBacking* _video_widget{nullptr};
    OverlayWidget* _overlay_widget{nullptr};
};

int main(int argc, char** argv)
{
    QApplication app(argc, argv);

    // Attach console on windows so we can see logging messages in terminal
#ifdef _WIN32
    if(AttachConsole(ATTACH_PARENT_PROCESS)) {
        FILE* stream = nullptr;
        freopen_s(&stream, "CONOUT$", "w", stdout);
        freopen_s(&stream, "CONOUT$", "w", stderr);
        freopen_s(&stream, "CONIN$", "r", stdin);
    }
#endif

    // Initialize gstreamer
    // DON'T FORGET: set environment variable "GST_PLUGIN_PATH" to load gstreamer plugins
    qDebug() << "Initializing GStreamer plugins (don't forget to set environment variable "
                "GST_PLUGIN_PATH)...";
    gst_init(nullptr, nullptr);
    guint major, minor, micro, nano;
    gst_version(&major, &minor, &micro, &nano);
    std::cout << "Successfully initialized gstreamer " << major << "." << minor << "." << micro
              << std::endl;

    MainWindow window;
    window.showNormal();
    auto result = app.exec();
    qDebug() << "Application exited with code: " << result;
    return result;
}

#include "Main.moc"

[SGaist]

Hi,

Shouldn't that overload of drawText do what you want ?

As for the black border, which border ?

[SeeRich]

@SGaist Thank you for looking at this. Are you suggesting that I should change my parameters being passed to drawText(...)?

I am wanting to get rid of the black box behind the overlay text. In this case, it should show yellow text on top of the white, yellow, and cyan vertical stripes from the video.

[SGaist]

@SeeRich yes, that overload should do what you want directly.

As for the black background, I suspect it's because you set the brush to NoBrush.

[SeeRich]

@SGaist thank you for your help. I guess I am a little confused.

Is the overload of drawText(...) that you are pointing me to supposed to help me fix the positioning (i.e. centered vertically and horizontally) or is it supposed to fix the black background around the overlay text?

As for the brush, what enumeration would you suggest? I didn't see one that's more equivalent to transparent which I believe is what I want in this case.

I believe the root cause of this issue is because GStreamer is rendering the video to the native window using the video widget's winId() function. It seems to me that the background isn't handled well because this is essentially bypassing Qt to render the video and the background that shows up behind the overlay text is what Qt believes is there (i.e. the MainWindow background).

What do you think?

[SeeRich]

@SGaist would it be helpful if I provided a CMakeLists.txt file to compile this example code?

[JoeCFD]

@SeeRich Use a QLabel(not QWidget) as your overlay widget and make it transparent. Then, the black background will be gone.

[SeeRich]

@JoeCFD Thank you for the suggestion! Unfortunately, it gave me the same result. I specifically changed the overlay code to be this:

class OverlayWidget : public QLabel
{
    Q_OBJECT
public:
    OverlayWidget(QWidget* parent) : QLabel(parent)
    {
        // setAttribute(Qt::WA_TranslucentBackground);
        // setWindowFlags(Qt::FramelessWindowHint | Qt::WindowStaysOnTopHint);
        setStyleSheet("background-color: transparent");
    }

Is this what you were suggesting?

[JoeCFD]

@SeeRich Yes, something like that. Can you try
setStyleSheet("background-color: green");
to see if the background color is green?

if the green color works, try the following:
QLabel * _overlayLabel{nullptr}; /* no need to create a overlay widget class */
_overlayLabel = new Qlabel( this );
_overlayLabel->setStyleSheet("border:none; background:transparent;");

[SeeRich]

@JoeCFD The green background did work; however, the _overlayLabel still has a black background. Could this be because Qt isn't aware of what gstreamer is rendering?

[JoeCFD]

@SeeRich your mainwindow has a black background stylesheet which affects its children. Assign its object name to its stylesheet and its children will not be affected anymore.

    MainWindow()
    {
        resize(720, 600);
        setAttribute(Qt::WA_StyledBackground);
        setObjectName( "mainWindiw" );
        setStyleSheet( QString( "MainWindow#%1 { background-color: black; }" ) .arg( objectName() );

[SeeRich]

@JoeCFD I really thought that was going to fix it, but unfortunately, same result.

[JoeCFD]

@SeeRich Change the background-color to blue to make sure the color from mainwindow stylesheet

    MainWindow()
    {
        resize(720, 600);
        setAttribute(Qt::WA_StyledBackground);
        setObjectName( "mainWindow" );
        setStyleSheet( QString( "MainWindow#%1 { background-color: blue; }" ) .arg( objectName() );

also try the following:

_overlayLabel = new Qlabel( this );
_overlayLabel->setAttribute(Qt::WA_StyledBackground);
_overlayLabel->setStyleSheet("border:none; background:transparent;");

[SeeRich]

@JoeCFD Sorry for the delay...

It is definitely the mainwindow background as the change to blue causes the text background box to change colors as well.

Setting the Qt::WA_StyledBackground attribute didn't change anything either.

[JoeCFD]

@SeeRich
can you change

        setAttribute(Qt::WA_StyledBackground);
        setObjectName( "mainWindow" );
        setStyleSheet( QString( "MainWindow#%1 { background-color: blue; }" ) .arg( objectName() );

to

        setAttribute(Qt::WA_StyledBackground);
        setObjectName( "mainWindow" );
        setStyleSheet( QString( "QWidget#%1 { background-color: blue; }" ) .arg( objectName() );

there is no MainWindow stylesheet. It should be QMainWindow.