QOpenGLWidget and Threads

[rtavakko]

Hey guys,

So I'm looking at the QOpenGLWidget class for reading and rendering video frames:

https://doc.qt.io/qt-5/qopenglwidget.html

Subclassing QOpenGLWidget and QOpenGLFunctions as suggested by the documentation:

class MyGLWidget : public QOpenGLWidget, protected QOpenGLFunctions
{
    ...
    void initializeGL() override
    {
        initializeOpenGLFunctions();
        glClearColor(...);
        ...
    }
    ...
};

My current implementation on the CPU consists basically of two threads reading OpenCV cv::Mat data from a cv::VideoCapture (https://forum.qt.io/topic/107895/qtconcurrent-vs-qthread-cpu-usage).

I'm thinking of having two instances of a VideoStream class each on a worker thread reading OpenCV frames and potentially OpenGL texture data from another application and sending them over to the GUI to be rendered on two display widgets (all of these would be subclassed the same way as the documentation suggests).

Communication would be done by passing the OpenGL ID of the texture to be displayed. It seems this is possible but this paragraph is a little confusing to me:

"Drawing directly to the QOpenGLWidget's framebuffer outside the GUI/main thread is possible by reimplementing paintEvent() to do nothing. The context's thread affinity has to be changed via QObject::moveToThread(). After that, makeCurrent() and doneCurrent() are usable on the worker thread. Be careful to move the context back to the GUI/main thread afterwards."

How would the GUI OpenGL context be shared with the threads?

Cheers!

[Kent-Dorfman]

My takeaway from the this (which is admittedly a WAG) is the context cannot be simultaneously shared by multiple worker threads, but only the one which recevied the latest moveToThread() may use the single "global" context.

[rtavakko]

@Kent-Dorfman Thanks for your reply! That does make sense and looking at the QOpenGLWidget threaded example it looks like that is what they're doing there using a worker thread. I think another alternative is to have multiple contexts on different threads sharign textures. Do you know by any chance how that could be done?

Cheers!

[Kent-Dorfman]

@rtavakko Unfortunately my opengl experience is old school immediate mode programming. I have yet to do anything real using the modern shaper/pipeline architecture. I think it's a given that the framework properly handles multiple QOpenGLWidgets, but as you've discovered, when you try to offload to worker threads there are some gotchas: the big one being that you now have to explicitly give the correct worker thread the context (and it probably cannot be simultaneously shared). I suspect that the modern pipeline architecture cannot handle multiple threads using the pipeline concurrently.

[wesblake]

I've got a similar question. I am trying to get our current (threaded) QGLWidget migrated to a QOpenGLWidget. So far based on an example it's successfully working meaning it runs, the glClear is working from the other thread, as is the "WTF" message so I know it's rendering at X fps.
What I can't figure out is how to get a QPainter to work!! I've tried everything. Our current QGLWidget based class renders QIMages at about 60 fps threaded, np using QPainter. I've read many things that seem to indicate I can still use QPainter in a separate thread as we are doing with QGLWidget, but I just can't get it to work. No matter how/where I create QPainter, it crashes. (cpp line 197)
Am I forced to use opengl directly? How would I draw a QImage like that? Thanks!

#ifndef GLWIDGET_H
#define GLWIDGET_H

#include "framequeue.h"

#include <QOpenGLWidget>
#include <QOpenGLContext>
#include <QImage>
#include <QThread>
#include <QMutex>
#include <QList>
#include <QWaitCondition>

class GLWidget;

class Renderer : public QObject
{
    Q_OBJECT

public:
    Renderer(GLWidget *);
    ~Renderer();
    void lockRenderer() { m_renderMutex.lock(); }
    void unlockRenderer() { m_renderMutex.unlock(); }
    QMutex *grabMutex() { return &m_grabMutex; }
    QWaitCondition *grabCond() { return &m_grabCond; }
    void prepareExit() { m_exiting = true; m_grabCond.wakeAll(); }

    FrameQueue *selfView;
    QList<FrameQueue*> remoteViewList;

signals:
    void contextWanted();

public slots:
    void start();
    void stop();
    void pause();
    void resume();
    void resumeToClose();
    void render();

private:
    static const int animationFPS = 30;
    static const int pipWidth = 160;
    static const int pipHeight = 120;

    bool m_inited;
    GLWidget *m_glwidget;
    QMutex m_renderMutex;
    QMutex m_grabMutex;
    QWaitCondition m_grabCond;
    bool m_exiting;

    bool pauseRendering;
};

class GLWidget : public QOpenGLWidget
{
    Q_OBJECT
public:
    GLWidget(QWidget *parent);
    ~GLWidget();
    void stopRendering();
    void startRendering();
    void pauseRendering();
    void resumeRendering();
    void resumeForClose();
    bool isStarted();

protected:
    void paintEvent(QPaintEvent *) Q_DECL_OVERRIDE { }

signals:
    void renderRequested();

public slots:
    void grabContext();

private slots:
    void onAboutToCompose();
    void onFrameSwapped();
    void onAboutToResize();
    void onResized();

private:
    QThread *m_thread;
    Renderer *m_renderer;
    bool m_isStarted;
};

#endif // GLWIDGET_H

#include "glwidget.h"

#include <QGuiApplication>
#include <QPainter>
#include <QDebug>

//Leak detection
#ifdef Q_OS_WIN32
#ifdef QT_DEBUG
#include "debugtools/reportinghook.h"
#include "debugtools/setdebugnew.h"
#define new DEBUG_NEW
#endif
#endif

GLWidget::GLWidget(QWidget *parent)
    : QOpenGLWidget(parent)
    , m_isStarted(false)
{
    connect(this, &QOpenGLWidget::aboutToCompose, this, &GLWidget::onAboutToCompose);
    connect(this, &QOpenGLWidget::frameSwapped, this, &GLWidget::onFrameSwapped);
    connect(this, &QOpenGLWidget::aboutToResize, this, &GLWidget::onAboutToResize);
    connect(this, &QOpenGLWidget::resized, this, &GLWidget::onResized);

    m_thread = new QThread;
    m_renderer = new Renderer(this);
    m_renderer->moveToThread(m_thread);
    connect(m_thread, &QThread::finished, m_renderer, &QObject::deleteLater);

    connect(this, &GLWidget::renderRequested, m_renderer, &Renderer::render);
    connect(m_renderer, &Renderer::contextWanted, this, &GLWidget::grabContext);
}

GLWidget::~GLWidget()
{
//    stopRendering();
}

void GLWidget::stopRendering()
{
    m_renderer->prepareExit();
    m_thread->quit();
    m_thread->wait();
    delete m_thread;
}

void GLWidget::startRendering()
{
    m_thread->start();
    m_isStarted = true;
    QMetaObject::invokeMethod(m_renderer, "start");
}

void GLWidget::pauseRendering()
{
    QMetaObject::invokeMethod(m_renderer, "pause");
}

void GLWidget::resumeRendering()
{
    QMetaObject::invokeMethod(m_renderer, "resume");
}

void GLWidget::resumeForClose()
{
    QMetaObject::invokeMethod(m_renderer, "resumeToClose");
}

bool GLWidget::isStarted()
{
    return m_isStarted;
}

void GLWidget::onAboutToCompose()
{
    //We are on the gui thread here. Composition is about to begin, wait until the render thread finishes.
    m_renderer->lockRenderer();
}

void GLWidget::onFrameSwapped()
{
    m_renderer->unlockRenderer();
    //Assuming a blocking swap, our animation is driven purely by the vsync in this example.
    emit renderRequested();
}

void GLWidget::onAboutToResize()
{
    m_renderer->lockRenderer();
}

void GLWidget::onResized()
{
    m_renderer->unlockRenderer();
}

void GLWidget::grabContext()
{
    m_renderer->lockRenderer();
    QMutexLocker lock(m_renderer->grabMutex());
    context()->moveToThread(m_thread);
    m_renderer->grabCond()->wakeAll();
    m_renderer->unlockRenderer();
}

Renderer::Renderer(GLWidget *w)
    : m_inited(false),
      m_glwidget(w),
      m_exiting(false),
      pauseRendering(false)
{
    selfView = new FrameQueue("selfView");
    long lineCount = 4;
    int iRet = HAPIGetPropertyInt(catPhone, propPhoneCallsLimit, &lineCount);
    QString remoteName;
    for(int i=0; i<lineCount; i++){
        remoteViewList.append(new FrameQueue(remoteName.sprintf("remoteView%i",i)));
    }
}

Renderer::~Renderer()
{
    if(selfView)
        delete selfView;
    for(int i=0; i<remoteViewList.size(); i++){
        FrameQueue *curQueue = remoteViewList.at(i);
        if(curQueue)
            delete curQueue;
    }
}

void Renderer::start()
{
    HAPIRegisterSelfView(selfView);
}

void Renderer::stop()
{
    HAPIUnregisterSelfView(selfView);
}

void Renderer::pause()
{
    pauseRendering = true;
    HAPIUnregisterSelfView(selfView);
}

void Renderer::resume()
{
    pauseRendering = false;
    HAPIRegisterSelfView(selfView);
}

void Renderer::resumeToClose()
{
    pauseRendering = false;
}

void Renderer::render()
{
    if(m_exiting)
        return;

    QOpenGLContext *ctx = m_glwidget->context();
    if(!ctx)//QOpenGLWidget not yet initialized
        return;

    //Grab the context.
    m_grabMutex.lock();
    emit contextWanted();
    m_grabCond.wait(&m_grabMutex);
    QMutexLocker lock(&m_renderMutex);
    m_grabMutex.unlock();

    if(m_exiting)
        return;

    Q_ASSERT(ctx->thread() == QThread::currentThread());

    //Make the context (and an offscreen surface) current for this thread.
    //The QOpenGLWidget's fbo is bound in the context.
    m_glwidget->makeCurrent();

    if(!m_inited) {
        m_inited = true;
//        initializeOpenGLFunctions();
    }

    glClearColor(0.27f, 0.26f, 0.26f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    if(pauseRendering){
        return;
    } else {
        //TODO HERE
        //How do I get QPainter here?! The context is moved yet I'm getting errors!
//            QPainter p(m_glwidget);

        qDebug() << "WTF";
    }

    //Make no context current on this thread and move the QOpenGLWidget's context back to the gui thread.
    m_glwidget->doneCurrent();
    ctx->moveToThread(qGuiApp->thread());

    //Schedule composition. Note that this will use QueuedConnection, meaning that update() will be invoked on the gui thread.
    QMetaObject::invokeMethod(m_glwidget, "update");
}

[wesblake]

Bump. I've done everything and can't figure this out. I'm about to try a non-threaded QOpenGLWidget because it's my only choice at this point but I'm afraid it will impact performance too much.

[rtavakko]

@wesblake The topics are not directly related so it would've probably helped to start a new one.

What error are you getting?

I'm coming more and more to the conclusion that a single thread is the way to go most of the time.

[wesblake]

@rtavakko said in QOpenGLWidget and Threads:

@wesblake The topics are not directly related so it would've probably helped to start a new one.

What error are you getting?

I'm coming more and more to the conclusion that a single thread is the way to go most of the time.

Sorry, seemed similar but I can move it. Here is the error:

[rtavakko]

@wesblake said in QOpenGLWidget and Threads:

QPainter p(m_glwidget);

Try replacing:

QPainter p(m_glwidget);

with:

QPainter p(this);

[wesblake]

Thanks. I found a known bug in QOpenGLWidget now preventing me from using it. :( So, starting over. New trial, a regular QWidget, worker thread to resize and composite frames first.

[rtavakko]

@wesblake Hmm. Which bug is this?

[wesblake]

https://bugreports.qt.io/browse/QTBUG-49657
Duplicated ticket says fixed but it's not. It occurs exactly as described.

[Christian Ehrlicher]

I'm pretty sure this is not your bug as long as you don't use qt5.6 and fullscreen mode. Please provide a compilable, minimal example to show your problem.