How to render 7 and more QGLWidgets in a way the GUI does not stutter

sayezz

Hey,
I have a GUI application that shows the video frame of many different cameras. Each camera is rendered in a separate QGLWidget. I can turn off and turn on the cameras. If the camera is on, you can see the camera video frame, if the camera is off, the QGLWidget is just black. As long as only 1 or 2 cameras are on everything is working fine, but when every camera is on the GUI gets slow and stutters. It looks like the application cant render the QGLWidgets fast enough.
This is how i have implemented it (in parts and pseudocode):
The Camera Object:
@
class Camera{
...
private Image myImage;
bool isCameraOn;

void updateFrame(){
myQGLWIdget->updateFrame(myImage);
}

void setMyImage(Image myImage){
this->myImage = myImage;
}

bool isOn(){
return isCameraOn;
}
}
@

The WorkerThread
@
class MyThread{

private vector<Camera> cameras;

signals:
void updateGui();

void run(){
while(true){
usleep(100000);

 for(int i = 0; i < cameras.size(); i++){
     // some interprocess communication to get the new frame        
     if(newFrameReceived){
          cameras.at(i).setMyImage(newImgFrame);
    }
 }

emit updateGui();
}
}

}
@

The Main Applikation
@
class Main{
private:
vector<Camera> cameras;

//Create Cameras, Thread and
... new MyThread(cameras);

// connect MyThread updateGui with Main myUpdate
public slot:
void myUpdate();

void myUpdate(){
for(int i = 0; i < cameras.size(); i++){
// some interprocess communication to get the new frame
if(camera.at(i).isOn()){
cameras.at(i).updateFrame();
}
}
}

}
@

Hope that is enough code to understand how i update the QGLWidget.

I cant imaging that QT is to slow to render that many QGLWidgets so I must do something wrong, but i dont know what.
One possibility to reduce the stuttering is to slow down the run Methode of myThread.

But that is the only idea i have.

Any suggestions would be great.

Here is a Screenshot of a part of the appication, just that you can imaging how it loos.
!http://luna-arts.de/others/misc/gui.png(gui)!

Thanks
Regards
Denis

SGaist

Hi,

Looking at your code it looks like it's your design that doesn't scale. If I understood correctly, you have a set of networked camera and you either fetch the images from them at regular interval or they send it. By the way which size are these images ? What format ?

To get started, you should rather have a worker object that is in charge to get the image from the camera. Once it has it, emit a signal with the data for the widget to update (Qt 5 QThread documentation for the worker object approach).

Then you can multiply the feeds and have independents viewers.

Also, did you consider creating a backend for QtMultimedia ? The rest would be already implemented for you e.g. QCameraViewFinder

Hope it helps

sayezz

Thanks for your reply.
The images are fetched in a regular interval, how ever it is possible that during this interval camera1 has send a new new frame and camera2 has not, so i would send the new frame from camera1 and the old one (that has already bin displayed) from camera2 to the main application to update the gui.

The size of the images is different, some cameras could be 640x480 while other cameras have a 1920x1080 resolution. The format is always RGB. Does the size matters? My camera Previews are very small (224x180), so the gui does not have to render the full image but only the small preview.

When I understand QThread correct i use already a worker Object. I have to possibilities, the first one is to emit a signal each time a new image arrived ( in worst case i have to emit seven signals in one while loop. The other possibility is to emit one signal at the each loop step and update all cameras.

I didnt look at QtMultimedia. The Custom QGLWidget to show the images is something I have to use. It wasn't code by myself. It does also a lot more then just rendering the image, but that is not interesting for my problem.

I tried to separte all the code that is necessary to understand how my image is rendered.
My Main Gui Object:
@
class MyWidget : public QWidget{
Q_OBJECT

private:
std::vector<CameraControlBox*> cameras;

public:
MyWidget::MyWidget(){
initCameraListner();
}

void MyWidget::initCameraListner(){
thread = new QThread;
cameraListner = new CameraListner(cameras);
cameraListner->moveToThread(thread);

connect(thread, SIGNAL(started()), cameraListner, SLOT(process()));
connect(cameraListner, SIGNAL(updateThread(/*parameters*/)), this, SLOT(updateThread(/*parameters*/)));

thread->start();

}

private slots:
void MyWidget::updateThread(/parameters/){
refreshSensor();
}

private:
void MyWidget::refreshSensor(){
for(unsigned int i = 0; i < cameras.size(); i++){
cameras.at(i)->renderLastState();
}
}
}
@

The Worker Thread Object:
@
class CameraListner : public QObject{
Q_OBJECT

private:
std::vector<CameraControlBox*> cameras;

public:
CameraListner::CameraListner(std::vector<CameraControlBox*> cameras){
this->cameras = cameras;
}
public slots:
void CameraListner::process(){

while (threadRuuning){
    usleep(100000);
    //usleep(33000);

    //+++++++ read Camera Messages ++++++++++++++++++++
    for(unsigned int i = 0; i < cameras.size(); i++){

if(cameras.at(i)->isCamOn()){
MyData::MyImage img;

if(interprocessCommunication.read(img) == NewData){
 cameras.at(i)->setImg(img);
 //int id = cameras.at(i).getId();
 //emit updateEachCameraSeparate(id);
}else{
 cameras.at(i)->setCamStatusNoSignal();
}
        }else{
            cameras.at(i)->setCamStatusOffline();
        }
    }
    emit updateThread( /*parameters*/); //update always all cameras with only signal
}

}

signals:
void updateThread( /parameters/);
}
@

The Camera Object:
@
class CameraControlBox : public QLabel{
Q_OBJECT

private:
MySensorQGLWidget* sensor;
MyData::MyImage img;
MyData::MyImage noSingalImg;
MyData::MyImage offlineImg;
MyData::MyImage notConnectedImg;
int state;
bool camOn;

public:
void CameraControlBox::setStreamMsg(MyData::MyImage img){
this->img = img;
state = 0;
}

void CameraControlBox::setCamStatusOffline(){
state = 2;
}

void CameraControlBox::setCamStatusOnline(){
state = 1;
}

void CameraControlBox::setCamStatusNoSignal(){
state = 1;
}

void CameraControlBox::renderLastState(){
if(state == 0){
if(camOn){
this->sensor->updateFrame(img);
state = 0;
}else{
state = 2;
}

    previouseState = 0;
}else if(state == 1){
    this->sensor->updateFrame(noSingalImg);

state = 1;
previouseState = 1;
}else if(state == 2){
if(previouseState !=2){
this->sensor->updateFrame(offlineImg);
state = 2;
previouseState=2;
}
}else if(state == 3){
...
}else{
...
}
}
}
@

The Custom QGLWidget that is not done by myself:
@
class MySensorQGLWidget : public QGLWidget{
Q_OBJECT

void updateFrame(MyData::MyImage &img){
...
// a bunch of stuff to render the data
//load frame into Video as Texture
glBindTexture(GL_TEXTURE_2D, VideoTextureID);
glTexImage2D( GL_TEXTURE_2D, 0, GL_RGB8, display_frame.size().width, display_frame.size().height, 0,
GL_BGR, GL_UNSIGNED_BYTE, display_frame.data);
...
updateGL();
...
}

void paintGL(){
// a bunch of stuff to render the data
...
}
}
@

agocs

Which Qt version on which platform? Is sync to vblank enabled? If so, you will likely to block in updateGL() for up to 16 ms for each QGLWidget which is definitely not ideal and may explain why everything starts to crawl once multiple QGLWidget's are taken into use.

sayezz

I'm using Qt 5.2.1 on Ubuntu 12.04 64x. Sync to vblank should be enabled. I can't disable it (maybe I just don't know the correct way how to do it).

Thanks to your advice I found a solution that must be the correct way in my opinion.
In my worker thread (CameraListner) I emit no signals anymore, i just set the new frame in my Camera (CameraControlBox ):

@
public slots:
void CameraListner::process(){

while (threadRuuning){
    usleep(100000);
    //usleep(33000);

    //+++++++ read Camera Messages ++++++++++++++++++++
    for(unsigned int i = 0; i < cameras.size(); i++){

if(cameras.at(i)->isCamOn()){
MyData::MyImage img;

if(interprocessCommunication.read(img) == NewData){
 cameras.at(i)->setImg(img);
 //int id = cameras.at(i).getId();
 //emit updateEachCameraSeparate(id);
}else{
 cameras.at(i)->setCamStatusNoSignal();
}
        }else{
            cameras.at(i)->setCamStatusOffline();
        }
    }
    //emit updateThread( /*parameters*/); // no emit anymore
}

}
//not needed anymore
//signals:
// void updateThread( /parameters/);
}
@

In my Main Application I use now a QTimer:
@
class MyWidget : public QWidget{
Q_OBJECT

private:
std::vector<CameraControlBox*> cameras;

public:
MyWidget::MyWidget(){
initCameraListner();
}

void MyWidget::initCameraListner(){
thread = new QThread;
cameraListner = new CameraListner(cameras);
cameraListner->moveToThread(thread);

connect(thread, SIGNAL(started()), cameraListner, SLOT(process()));

thread->start();

QTimer *timer = new QTimer(this);
connect(timer, SIGNAL(timeout()), this, SLOT(refreshSensor()));
timer->start(40);

}
@

Thats working much better but not perfect. Now I can use up to 4 cameras without noticing to many lags, when I add more cameras the gui starts stuttering again. Maybe I have to check my code again and see if I have some other mistakes, because I think using QTimers must be the correct way.

I have one more question:
I have more than one method to update things in my GUI:
refreshSensors() : refresh the sensors
refreshLabels() : refresh some labels
refreshStatusBar() : refresh Statusbar (e.g. heart beat to indicate that thread is running)

Is it correct to use the QTimer like this:
@
QTimer *timer = new QTimer(this);
connect(timer, SIGNAL(timeout()), this, SLOT(refreshSensor()));
connect(timer, SIGNAL(timeout()), this, SLOT(refreshLabels()));
connect(timer, SIGNAL(timeout()), this, SLOT(refreshStatusBar()));
timer->start(40);
@

or should it be something like this:
@
QTimer *timer = new QTimer(this);
connect(timer, SIGNAL(timeout()), this, SLOT(refreshAll()));
timer->start(40);

void refreshAll(){
refreshSensors() ;
refreshLabels();
refreshStatusBar();
}
@

or does it not care at all?

What I'm going to try no is to remove all GUI-relevant signals that come from other classes, so the MainGUI Object is the only one that triggers the GUI repaint using QTimer. Maybe that will reduce the stuttering.

sayezz

Hey, one more Question...
I have measured the timings using the QTimer from my last post:
When not rendering any QGLWIdget the methode takes 0.001ms
When I render only one QGLWidget it takes about 0.7 ms
When I render two QGLWidgets it takes about 17.4 ms
When I render three QGLWidgets it takes about 48.8 ms
When I render four QGLWidgets it takes about 64.7 ms
When I render five QGLWidgets it takes about 81.2 ms

Any Idea why there is such a huge jump from two Widgets to three Widgets?
Any Idea how I can improve the rendering of a QGLWidget?