Best way to render video frame in Qt from an IP Camera.

SaiyanRiku

Hi,

I'm new in Qt and i need to render an MJPEG stream from a IP Camera.

I got the JPEG frame from the stream and i decompress it with libjpeg-turbo to improve performance. I built a QImage from the data and a draw it in a QWidget.
This is globally fast on a powerfull machine like my linux 3.2 64 bits (LMDE) intel core i5, but inside a VirtualBox WM over Windows 7 32 bits it's not, it takes me 90-100% of my CPU time to render only one stream at 25 FPS, since VLC use only 35-40% of CPU time.

I Google my problem and I found many proposition i tried:

Use a QWidget (the solution with the best performance i found)
Use a QGraphicsView and QGraphicsScene (same as QVideoWidget)
Use a QGLWidget (same a QWidget if video acceleration is not available, but catastrophic if not)
Use a QAbstractVideoSurface from QtMultimedia (http://qt-project.org/doc/qt-4.8/multimedia-videowidget.html) (no advantage in front of QWidget)
Use Phonon (i don't try but seems to be too much high level, i need to control each frame i process)
Render directly on X11, and equivalent on Windows (GDI+?) (don't try but not my favorite solution since it's less portable)

Considering all theses solutions (or maybe others i don't know), which one do you think is supposed to be the good one?

From my test the best one is QWidget, but it's not what all discussion about performance says.

I read all topics I found about performance and I tried them, such as:

Enable/disable some feature on the widget:
** setAutoFillBackground(false);
** setAttribute(Qt::WA_NoSystemBackground, true);
** setAttribute(Qt::WA_PaintOnScreen, true);
Use only an RGB32 image (http://qt-project.org/doc/qt-4.8/QPainter.html#performance)
Pre-render QImage in separate thread (http://labs.qt.nokia.com/2010/01/21/qt-graphics-and-performance-generating-content-in-threads/)
These features improve really the rendering (especially on Linux), but it's not enough (especially on Windows).

Qt::WA_PaintOnScreen or QDirectPainter seems to work only on linux is there an equivalent on Windows?

Please, what can I do to increase the performance in Qt for my video rendering?

See below some tests I did on QT. This consist to render 30 JPEG images where original size is 1280x800 to a specified size. The image is pre-scaled at decompression by selecting the best scale ratio.
D=Decompression time (libjpeg-turbo part)
L=Pixmap loading time (pixmap to GUI image format (QImage))
S=Scaling time
X=Rendering time

1 ) Using an Intel core i5 on Linux 3.2 64 bits (LMDE): Test with libjpeg-turbo (decompression is TJPF_BGRA, with TJFLAG_FASTUPSAMPLE flags)
[Qt +TurboJPEG ] Test 1: (30 images 1280x800 => 160x100 (160x100)) is 72(85) ms => D=30 ms, L=0 ms, S=0 ms, X=38 ms
[Qt +TurboJPEG ] Test 2: (30 images 1280x800 => 320x200 (320x200)) is 138(151) ms => D=43 ms, L=0 ms, S=0 ms, X=86 ms
[Qt +TurboJPEG ] Test 3: (30 images 1280x800 => 640x400 (640x400)) is 197(209) ms => D=60 ms, L=0 ms, S=0 ms, X=113 ms
[Qt +TurboJPEG ] Test 4: (30 images 1280x800 => 1280x800 (1280x800)) is 358(370) ms => D=91 ms, L=0 ms, S=0 ms, X=246 ms
[Qt +TurboJPEG ] Test 5: (30 images 1280x800 => 320x240 (320x200)) is 90(100) ms => D=54 ms, L=0 ms, S=0 ms, X=30 ms
[Qt +TurboJPEG ] Test 6: (30 images 1280x800 => 420x360 (640x400)) is 120(129) ms => D=60 ms, L=0 ms, S=0 ms, X=30 ms
[Qt +TurboJPEG ] Test 7: (30 images 1280x800 => 640x480 (640x400)) is 135(146) ms => D=60 ms, L=0 ms, S=0 ms, X=36 ms
[Qt +TurboJPEG ] Test 8: (30 images 1280x800 => 1024x780 (1280x800)) is 252(266) ms => D=90 ms, L=0 ms, S=30 ms, X=110 ms
[Qt +TurboJPEG ] Total redering time 1456 ms

Using a Windows 7 32 bits inside Virtual Box on the same machine: Test with libjpeg-turbo (decompression is TJPF_BGRA, with TJFLAG_FASTUPSAMPLE flags)
[Qt +TurboJPEG ] Test 1: (30 images 1280x800 => 160x100 (160x100)) is 70(70) ms => D=50 ms, L=20 ms, S=0 ms, X=0 ms
[Qt +TurboJPEG ] Test 2: (30 images 1280x800 => 320x200 (320x200)) is 101(101) ms => D=91 ms, L=0 ms, S=0 ms, X=10 ms
[Qt +TurboJPEG ] Test 3: (30 images 1280x800 => 640x400 (640x400)) is 170(170) ms => D=140 ms, L=0 ms, S=0 ms, X=30 ms
[Qt +TurboJPEG ] Test 4: (30 images 1280x800 => 1280x800 (1280x800)) is 660(660) ms => D=570 ms, L=0 ms, S=0 ms, X=90 ms
[Qt +TurboJPEG ] Test 5: (30 images 1280x800 => 320x240 (320x200)) is 131(131) ms => D=101 ms, L=0 ms, S=30 ms, X=0 ms
[Qt +TurboJPEG ] Test 6: (30 images 1280x800 => 420x360 (640x400)) is 290(290) ms => D=200 ms, L=0 ms, S=80 ms, X=10 ms
[Qt +TurboJPEG ] Test 7: (30 images 1280x800 => 640x480 (640x400)) is 390(390) ms => D=210 ms, L=0 ms, S=160 ms, X=20 ms
[Qt +TurboJPEG ] Test 8: (30 images 1280x800 => 1024x780 (1280x800)) is 1031(1031) ms => D=601 ms, L=0 ms, S=400 ms, X=30 ms
[Qt +TurboJPEG ] Total redering time 2843 ms

The scale time on windows is very long.

How can I improve the scale time of my image?

Thanks you very much for you help.

DerManu

Is 2D (and maybe 3D) graphics acceleration enabled in the VBox settings of the machine?
Do you use Qt::FastTransformation when scaling the image?

Apart from that, there simply are limitations to what such a machine can do. VLC gets its speed by using graphics acceleration, i.e. OpenGL overlay. You should be able to get pretty much the same performance with using the opengl graphicssystem (pass -graphicssystem opengl on the command line to your app, or call QApplication::setGraphicsSystem("opengl"); before creating the QApplication in main.) and then not using a QImage (because it always uses raster if I remember correctly), but QPixmap. Or even better, try to get your image data from your jpeg decompression library directly into an OpenGL texture, you'll need non-Qt low level code for that, though.

And if this is really serious business: VLC is free software, look at how they're doing it! (I'm guessing ffmpeg -> openGL texture)

SaiyanRiku

Thank you for your fast answer.

The fact is if I set -graphicssystem opengl or I use QGlWidget on Virtual Box, is it very slow. Yes! 3D acceleration is not enabled, but VLC works fine.

I can't really use QPixmap since I decode the JPEG image from another and it is not allowed to use QPixmap outside GUI Thread. If i decode all frame in GUI thread, the UI become less responsive.

Nevertheless, i ask to the VLC team, you right! They use ffmpeg or SDL_Image to decode the stream, and XVideo with MIT-SHM, OpenGL or DirectX for rendering (http://mailman.videolan.org/pipermail/vlc-devel/2012-June/088716.html). More over the rendering is not done in the GUI thread, but from another thread with direct access to these API.

I will try DirectX and XVideo, and report the feedback.

SaiyanRiku

I just report the feedback for every one need this:

I tried DirectX and i found that it is better to use DirectX to directly stretch and render the frame (using BltBit function).

? Offline

Hi All,

Sorry to reopen this post. I wanted to know whether the current Qt libraries handle the rendering of images in a camera.

Ansif

SaiyanRiku

I think you can by using Qt Multimedia (http://qt-project.org/doc/qt-5/multimediaoverview.html)

The doc said :

bq. Use a camera, including viewfinder, image capture, and movie recording

Nevertheless, you can be limited by features and supported codecs, it depends what you really need to do.

mrdebug

You can use a easy approach without to use componets.
Mjpg is a jpeg stream. Each image starts with 0xff 0xd9 and ends with 0xff 0xd8.

? Offline

[quote author="mrdebug" date="1400594886"]You can use a easy approach without to use componets.
Mjpg is a jpeg stream. Each image starts with 0xff 0xd9 and ends with 0xff 0xd8.
[/quote]

Hello,
This is what I am really looking for. Thank you for your help. Can you explain me little more about this and also how can we do a fast rendering from a camera.

Ansif