Live camera image in QGraphicsVIew

JulienMaille

Dear all, I have a widget that displays the live image sent by a camera.
It worked well so far with 1.3MPixel images @ 30fps but it's getting laggy now that I have higher resolutions.
This is some code I've done a while ago and I'm now wondering if what I did is optimal.

I have a camera thread with a double buffer storing incoming images and emitting a signal to the main thread.
It is connected to a slot where I convert my uchar* buffer into a QPixmap and call QGraphicsPixmapItem::setPixmap().

If you guys think of any web resource where I could find some hints on how to do this in a better way, that would be great. I'm sure this is a common problem and there must be "good practice" out there, but I was not able to find them.

Remark: I would like to stick with QGraphicsView, since QGraphicsItems are very helpful to display overlays and so on.

SGaist

Hi,

One thing could be against you is that depending on what image format you get from your camera, there is a conversion to do that might be time consuming (also the bandwidth usage rises significantly)

Also, how do you get the images ? USB ? Network ?

JulienMaille

I'm reading 8bit images from USB. There is no conversion involved apart from filling a QImage from a uchar* buffer.

SGaist

Are you using a QImage that makes a deep copy of the data ?

JulienMaille

Yes:
@void MainWindow::slotDisplayLiveImage(const uchar* buffer, int imgW, int imgH)
{
if( !buffer ) return;
QImage image(buffer, imgW, imgH, imgW, QImage::Format_Indexed8);
image.setColorTable(table);

liveScene->setImage(QPixmap::fromImage(image));

}@
To be honest, I'm a bit lost on the general strategy: ie. should I have a producer thread that emit signal to display images from the main thread.
Or a producer thread and a consumer thread.
How do I handle when images are produced faster that they are displayed.
How do I easily secure the "current display" buffer and sync that between thread.

There must be some general guidelines somewhere as I am obviously not the first one to do that!

SGaist

You might also consider using OpenGL

JulienMaille

Does that answer one of my questions?

SGaist

Directly ? No. But since you have lots of image data to show, OpenGL might be one solution to speed things up.

The consumer/producer solution is good. You could use a FIFO and drop images if your consumer is not fast enough. Using Qt's implicit shared class (QByteArray/QImage but with a deep copy of the data) you won't have to worry about securing the data, only the access to the FIFO.

Hope this helps better

JulienMaille

I hope it does not sound disrespectful but it does not help a lot.
I was in fact looking for guidelines from people who may have already been working on the same kind of stuff.
"Use FIFO and drop images", ok why not but you know what they say "devil is in the details", how do you limit the size of your FIFO, how do you decide to drop images, etc.

SGaist

The first thing to do is to identify the bottlenecks you have (i.e the conversion from QImage to QPixmap) using either a QTime for basic measurement or a more professional tool if you have one.

Then you have two model applicable to your problem:

push
pull

Since you are considering the pull model (with the consumer/producer paradigm) you can use this design:

A ring buffer with pre-allocated buffers where you copy the image data and your other thread gets them as fast as it can.
A semaphore to make your consumer waits when the ring buffer is empty.

The ring buffer allows you to easily know when your system cannot sustain the throughput.
It's then up to you to decide whether you overwrite the old data, stop the stream, emit a warning etc... That is essentially application dependent

If you need to display things faster, then you probably have to go the OpenGL way (there is a QQ article about threaded rendering)

One last point of interest might be to get the data as fast as possible to the graphic card using i.e. CUDA, OpenCL etc...

I would need more information about what your application does to give better advices.

JulienMaille

Thanks a lot for your answer.
Bottleneck depends on camera/system so I should be able to handle slow framerates (images are consumed faster that produced) and high framerate/slow computer (images are produced faster than consumed)

cuda/hardware acceleration is not an option here unfortunately

when the consumer thread ask for an image, how would you then pass it to the mainthread for display? and lock it until it is actually drawn?

SGaist

For the slow frame rates your consumer will wait on the producer (either QWaitCondition or QSemaphore) so that should no really be a problem, the screen update will happen just after a new image has arrived.

For the other case, you can reduce the size of the image shown, or the image produced (if possible to configure the camera) or slow down the frame rate either configuring the camera or the producer (i.e dropping one frame out of 5 or whatever it would take)

Copy the data from the ring buffer (i.e in a QByteArray) and you don't need to lock it any further (besides while copying of course).

JulienMaille

Thanks (again) for the details.
My current implementation has a ring buffer however there's no consumer, just a producer that emits signal when an image is ready.
The flaw here is that I don't lock anything when the main thread handles the signal (ie. copy the data)

Now it's time to code. Thank you SGaist!

SGaist

What's the signal signature and how does you application react to it ?

JulienMaille

The signal passes the buffer "pointer (do not laugh!), width and height of the image":http://qt-project.org/forums/viewthread/29786/#134745.
I know I should be ashamed to say that, but I wrote that a while ago and never touched it since then.

SGaist

Just to be sure:

You get an image

Add it to your ring buffer

emit a signal with the address of the data in the ring buffer

Did I understand you right ?

JulienMaille

It's supposed to be a bit more secure.
I have a ping/pong buffer plus a third "locked" buffer that should prevent concurrent access, but that is in part broken and that's why I ended up here asking for guidelines :)

SGaist

Ok ok,

Then with the ring buffer you can have another thread read the data in a QByteArray or directly in a QImage and send that to your gui thread. A bit like the mandelbrot example

JulienMaille

I'll have a look at that example. Thanks again