QCamera: draw on Viewfinder

StarterKit

Hi all,
I do trivial task - to read image from camera and then QR code from it.
Draft example works well and I see that quality of QR recognition depends on QR position in the image. So, I would like to cut a square from the image and process only it. But I need to show this square to the user in a viewfinder.
Currently my viewfinder is simple QVideoWidget. And I haven't found any way to draw on it. I searched on the Internet, found some advices but I'm not convinced and ready to take some of them:

one option is to make a descendant of QAbstractVideoSurface and paint on it, but it looks quite complicated way to simply draw four lines.
another proposal was to put a transparent widget on top of QVideoWidget and draw on it. But I'm not sure that I'll not mix coordinates of widget with coordinates of image.

Does someone have experience of doing similar task? May you give me some advice?

eyllanesc

@StarterKit PyQt5 or PySide2?

StarterKit

@eyllanesc said in QCamera: draw on Viewfinder:

@StarterKit PyQt5 or PySide2?

PySide2 (5.15), plan to move to PySide6 as soon as 6.2 will be released.

StarterKit

Ok, finally I'm moving to PySide6 with Qt 6.2.
I have a small example that works with Camera and shows video in QVideoWidget using code like this:

self.viewfinder = QVideoWidget(self)
self.camera = QCamera()
self.captureSession = QMediaCaptureSession()
self.captureSession.setCamera(self.camera)
self.captureSession.setVideoOutput(self.viewfinder)

It works ok, I see live video on a screen.

Now I want to draw something on top of 'viewfinder' (QVideoWidget).
I read several sources (I found this one the most useful) and decided to create an overlay widget. I did it and it works. At least it works when I use QLabel or QButton as a parent for overlay widget. I can drow something above them in this case.
But I have nothing when I try to use QVideoWidget as a parent. It appears that QVideoWidget draws on top of everything by its own.

So, the question remains - how to draw a couple of lines on top of video frame displayed to a user in QVideoWidget?

StarterKit

@eyllanesc I found your example on Stackoverflow - it is nice and handy. Thanks a lot.

But I struggle to set size of _videoitem . I can make _gv any size I wish. But then _videoitem takes only small area within it. May you give me a hint how to expand _videoitem to entire _gv area?

StarterKit

Ok, I manage to re-scale it with below code:

    def resizeEvent(self, event):
        bounds = self._scene.itemsBoundingRect()
        self._gv.fitInView(bounds, Qt.KeepAspectRatio)
        self._gv.centerOn(0, 0)
        self._gv.raise_()

But it is triggered only when I manually resize the window. It doesn't work while camera isn't started and I can't find a good event to make a resize after camera start only once.

eyllanesc

@StarterKit Since you are working in Qt6 and in that new version the QtMultimedia API has changed, now it is easier to draw on a QVideoWidget (or other output such as QVideoGraphicsItem, VideoOutput QML item), in a few days I will open a blog about a post on that topic but you can see something similar in this C++ answer: https://stackoverflow.com/questions/69432427/how-to-use-qvideosink-in-qml-in-qt6/69432938#69432938 or gist https://gist.github.com/eyllanesc/6486dc26eebb1f1b71469959d086a649

StarterKit

@eyllanesc ok, thanks.
I solved my task with QVideoGraphicsItem and its nativeSizeChanged signal, it works fine, but I'll take a look on your articles later.

Here is my working example if anyone interested. Not an ideal code but it is pretty simple and you may adopt it for your needs.
It simply starts camera and displays you a preview with bounding square on top of it. Then it constantly captures an image from camera, crops this inner square and tries to read a QR code from it. Content of QR code is printed to console. Nothing more but it is only a piece of code as example.

import io
from pyzbar import pyzbar
from PIL import Image, UnidentifiedImageError
from PySide6.QtCore import Qt, Signal, QBuffer, QRectF
from PySide6.QtGui import QImage, QPen
from PySide6.QtWidgets import QApplication, QMainWindow, QWidget, QVBoxLayout, QGraphicsScene, QGraphicsView
from PySide6.QtMultimedia import QCamera, QMediaCaptureSession, QImageCapture
from PySide6.QtMultimediaWidgets import QGraphicsVideoItem


#----------------------------------------------------------------------------
class CameraWnd(QMainWindow):
    QR_SIZE = 0.75   # Size of rectangle for QR capture
    next_ready = Signal(bool)

    def __init__(self):
        QMainWindow.__init__(self)
        self.processing = False
        self.rectangle = None
        self.layout = QVBoxLayout()

        self.scene = QGraphicsScene(self)
        self.view = QGraphicsView(self.scene)
        self.viewfinder = QGraphicsVideoItem()
        self.view.setMinimumSize(720, 405)
        self.view.setFrameStyle(0)
        self.scene.addItem(self.viewfinder)
        self.layout.addWidget(self.view)

        self.wnd = QWidget(self)
        self.wnd.setLayout(self.layout)
        self.setCentralWidget(self.wnd)

        self.camera = QCamera()
        self.captureSession = QMediaCaptureSession()
        self.img_capture = QImageCapture(self.camera)

        self.captureSession.setCamera(self.camera)
        self.captureSession.setVideoOutput(self.viewfinder)
        self.captureSession.setImageCapture(self.img_capture)

        self.camera.errorOccurred.connect(self.on_cam_error)
        self.next_ready.connect(self.on_ready)
        self.img_capture.errorOccurred.connect(self.on_error)
        self.img_capture.readyForCaptureChanged.connect(self.on_ready)
        self.img_capture.imageCaptured.connect(self.captured)
        self.viewfinder.nativeSizeChanged.connect(self.video_size_changed)

        self.camera.start()

    def video_size_changed(self, _size):
        self.resizeEvent(None)

    # Take QImage or QRect (object with 'width' and 'height' properties and calculate position and size
    # of the square with side of self.QR_SIZE from minimum of height or width
    def calculate_center_square(self, img_rect) -> QRectF:
        a = self.QR_SIZE * min(img_rect.height(), img_rect.width())   # Size of square side
        x = (img_rect.width() - a) / 2         # Postion of the square inside rectangle
        y = (img_rect.height() - a) / 2
        if type(img_rect) != QImage:   # if we have a bounding rectangle, not an image
            x += img_rect.left()       # then we need to shift our square inside this rectangle
            y += img_rect.top()
        return QRectF(x, y, a, a)

    def resizeEvent(self, event):
        bounds = self.scene.itemsBoundingRect()
        self.view.fitInView(bounds, Qt.KeepAspectRatio)
        if self.rectangle is not None:
            self.scene.removeItem(self.rectangle)
        pen = QPen(Qt.green)
        pen.setWidth(0)
        pen.setStyle(Qt.DotLine)
        self.rectangle = self.scene.addRect(self.calculate_center_square(bounds), pen)
        self.view.centerOn(0, 0)
        self.view.raise_()

    def on_error(self, _id, _error, error_str):
        print(f"Error: {error_str}")
        self.processing = False

    def on_cam_error(self, _error, error_str):
        print(f"Error: {error_str}")

    def on_ready(self, ready: bool):
        if ready and not self.processing:
            self.img_capture.capture()
            self.processing = True

    def captured(self, _id: int, img: QImage):
        self.decode(img)
        self.processing = False
        self.next_ready.emit(self.img_capture.isReadyForCapture())

    def decode(self, qr_image: QImage):
        cropped = qr_image.copy(self.calculate_center_square(qr_image).toRect())

        buffer = QBuffer()
        buffer.open(QBuffer.ReadWrite)
        cropped.save(buffer, "BMP")
        try:
            pillow_image = Image.open(io.BytesIO(buffer.data()))
        except UnidentifiedImageError:
            print("Image format isn't supported")
            return
        barcodes = pyzbar.decode(pillow_image, symbols=[pyzbar.ZBarSymbol.QRCODE])
        if barcodes:
            print(f"Decoded QR: {barcodes[0].data.decode('utf-8')}")


#----------------------------------------------------------------------------
def main():
    app = QApplication()

    camera = CameraWnd()
    camera.show()

    return app.exec()


#----------------------------------------------------------------------------
if __name__ == '__main__':
    main()

eyllanesc

@StarterKit I have added an example of how you can process the image using pyzbar. At the moment PySide6 6.2.0 has a bug so the example is written in PyQt6 but for the next release it should be fixed. https://gist.github.com/eyllanesc/6486dc26eebb1f1b71469959d086a649#gistcomment-3920960

StarterKit

@eyllanesc I tried your code but somehow it fails with AttributeError: 'PySide6.QtMultimedia.QVideoFrame' object has no attribute 'bits' at line 14 of utils.py. At the same time I see that QVideoFrame has bits method indeed. Looks like some strange error, probably in PySide6.

Also I think your method is good for some complex processing of video frames before displaying. For my humble task it is an overkill. Anyway, thanks a lot for your example.

eyllanesc

@StarterKit I have pointed out in my comment: At the moment PySide6 6.2.0 has a bug so the example is written in PyQt6 but for the next release it should be fixed., See https://bugreports.qt.io/browse/PYSIDE-1674. So if you want to use PySide6 6.2.0 you must apply the patch this associated with the report and recompile PySide6. I did it and therefore I provided that tested example applying that patch.

Looking at your example I just saw that you only wanted to get the frame and process it, I thought that after processing it you wanted to modify the image and publish it in the same output. If so then in Qt5 you could implement a custom QAbstractVideoSurface as I show in the following example: https://stackoverflow.com/questions/67082179/qvideowidget-content-isnt-grabed-from-widget/67082564#67082564. It confuses me what you point out in your initial post and what your code does: but it looks quite complicated way to simply draw four lines.

StarterKit

@eyllanesc thanks for your reference and information about the bug.
Yes, my task isn't so complex, I really need only to highlight the area of QR square cropping.
Now with more knowledge I understand that my question wasn't precise enough - but at that time I thought differently. Thank you for help.