Creating a Video Widget with PyQt5

bluu · wrote on 4 Nov 2016, 23:08

Hi there,

I'm trying to get a video viewer widget which I developed using PyQt5 in Windows to work on Linux 16.04.

The video widget called ViewfinderWidget subclasses QWidget, including a surface, VideoWidgetSurface, which implements QAbstractVideoSurface. The video viewer takes in frames from a camera (a Point Grey Flea3 camera) in numpy array format and gives these to the surface to present.The implementation is mainly inspired by this tutorial.

Everything works fine and rosy with a test script on Windows, but on the linux version, the app opens with a black screen in the viewer widget. Furthermore, I receive the following errors:

QWidget::paintEngine: Should no longer be called
QWidget::paintEngine: Should no longer be called
QWidget::paintEngine: Should no longer be called
QPainter::begin: Paint device returned engine == 0, type: 1
QPainter::worldTransform: Painter not active
QPainter::setWorldTransform: Painter not active

This happens every image grab loop.

I traced the calls with a debugger, and I have determined that this problem has nothing to do with the camera or frame grabbing...those give correct frames that change as I move the camera. The problem seems to be that the QPainter isn't working correctly for VideoWidgetSurface. When I evaluate 'QPainter.isActive()' it returns false in the loop.

I've read some other people's errors similar to mine, but their main error seems to be that paintEvent is called directly, or that drawing occurs outside of the paintEvent. My project doesn't seem to be doing any of those things.

What are some things I should try to get things working?

Any help would be hugely appreciated.

See my code below:

"""
This class implements the specifications for QAbstractVideoSurface
"""
class VideoWidgetSurface(QAbstractVideoSurface):

    def __init__(self, widget):
        self.widget = widget
        # default is a blank screen
        self.currentFrame = QVideoFrame()
        self.targetRect = QRect()
        super().__init__()

    """
    From the supportedPixelFormats() function we return a list of pixel formats the surface can paint. The order of the list hints at which formats are preferred by the surface.
    These are the Flea-3's supported formats

    Since we don't support rendering using any special frame handles we don't return any pixel formats if handleType is not QAbstractVideoBuffer::NoHandle.
    """
    def supportedPixelFormats(self, type=None):
        if(type == QAbstractVideoBuffer.NoHandle):
            return [QVideoFrame.Format_RGB24,
                    QVideoFrame.Format_Y8,
                    QVideoFrame.Format_Y16,
                    QVideoFrame.Format_YUV444,
                    QVideoFrame.Format_CameraRaw]
        else:
            return []

    """
    In isFormatSupported() we test if the frame type of a surface format maps to a valid QImage format,
    that the frame size is not empty, and the handle type is QAbstractVideoBuffer::NoHandle.
    Note that the QAbstractVideoSurface implementation of isFormatSupported() will verify that the list of supported
    pixel formats returned by supportedPixelFormats(format.handleType()) contains the pixel format and that the size is
     not empty so a reimplementation wasn't strictly necessary in this case.
    """
    def isFormatSupported(self, format):
        imageFormat = QVideoFrame.imageFormatFromPixelFormat(format.pixelFormat())
        size = format.frameSize()

        return imageFormat != QImage.Format_Invalid and not size.isEmpty() and format.handleType() == QAbstractVideoBuffer.NoHandle

    def start(self, format):
        imageFormat = QVideoFrame.imageFormatFromPixelFormat(format.pixelFormat())
        size = format.frameSize()

        if (imageFormat != QImage.Format_Invalid and not size.isEmpty()):
            self.imageFormat = imageFormat
            self.imageSize = size
            self.sourceRect = format.viewport()

            super().start(format)

            self.widget.updateGeometry()
            self.updateVideoRect()

    def updateVideoRect(self):
        size = self.surfaceFormat().sizeHint()
        size.scale(self.widget.size().boundedTo(size), Qt.KeepAspectRatio)

        self.targetRect = QRect(QPoint(0, 0), size)
        self.targetRect.moveCenter(self.widget.rect().center())

    def present(self, frame):
        # print(self.surfaceFormat().pixelFormat())
        # print(frame.pixelFormat())
        if(self.surfaceFormat().pixelFormat() != frame.pixelFormat() or self.surfaceFormat().frameSize() != frame.size()):
            self.setError(self.IncorrectFormatError)
            self.stop()
            return False
        else:
            self.currentFrame = frame
            self.widget.repaint(self.targetRect)
            return True

    def paint(self, painter):
        if (self.currentFrame.map(QAbstractVideoBuffer.ReadOnly)):
            oldTransform = painter.transform()

            if self.surfaceFormat().scanLineDirection() == QVideoSurfaceFormat.BottomToTop:
                painter.scale(1,-1)
                painter.translate(0, -self.widget.height())

            image = QImage(self.currentFrame.bits(),
                           self.currentFrame.width(),
                           self.currentFrame.height(),
                           self.currentFrame.bytesPerLine(),
                           self.imageFormat)

            painter.drawImage(self.targetRect, image, self.sourceRect)
            painter.setTransform(oldTransform)
            self.currentFrame.unmap()
        else: #if invalid frame received
            painter.drawText(self.targetRect, Qt.AlignCenter, "No feed...\nCheck Camera Status")


    def stop(self):
        self.currentFrame = QVideoFrame()
        self.targetRect = QRect()
        super().stop()
        self.widget.update()


    def videoRect(self):
        return self.targetRect


"""
The ViewfinderWidget class uses the VideoWidgetSurface class to implement a video widget.
"""
class ViewfinderWidget(QWidget):
    """"
    This class is the widget that holds the viewport into the camera
    """
    # time in ms to wait before restarting the stream
    RESTART_TIME = 1

    erroredOut = pyqtSignal(str, ErrorPriority)

    def __init__(self, parent=None):
        super().__init__(parent)
        self.setAutoFillBackground(False)
        self.setAttribute(Qt.WA_NoSystemBackground, True)
        self.setAttribute(Qt.WA_PaintOnScreen, True)
        palette = self.palette()
        palette.setColor(QPalette.Background, Qt.black)
        self.surface = VideoWidgetSurface(self)
        self.lastShowTime = time.time()

        # rate per second at which camera view is refreshed and re-drawn (since drawing takes time)
        self.FRAME_RATE = 24.0  # anything above 24 is considered smooth motion

    def closeEvent(self, QCloseEvent):
        del self.surface

    def videoSurface(self):
        return self.surface

    def sizeHint(self):
        return self.surface.surfaceFormat().sizeHint()

    def paintEvent(self, event):
        painter = QPainter(self)

        if(self.surface.isActive()):
            videoRect = self.surface.videoRect()

            if(not videoRect.contains(event.rect())):
                region = event.region()
                region.subtracted(QRegion(videoRect))

                brush = self.palette().window()

                for rect in region.rects():
                    painter.fillRect(rect, brush)

            self.surface.paint(painter)
        else:
            painter.fillRect(event.rect(), self.palette().window())

    def resizeEvent(self, event):
        super().resizeEvent(event)
        self.surface.updateVideoRect()

    @pyqtSlot(np.ndarray)
    def processFrame(self, frame):
        """
        Process the Numpy array from a camera to display it on the surface
        :param frame: the numpy pixel array from the camera daemon
        :return:
        """

        # this updates the video display according to the frame rate of the camera, otherwise it is unnecessary to update the feed any faster
        if (time.time() - self.lastShowTime < 1.0/self.FRAME_RATE):
            return

        # construct QImage
        qIm = QImage(frame, frame.shape[1], frame.shape[0], QImage.Format_RGB888)

        # construct a video frame from the QImage
        vidFrame = QVideoFrame(qIm)
        vidFrame.map(QAbstractVideoBuffer.ReadOnly)

        # present on video surface
        self.surface.present(vidFrame)

        self.currImg = qIm

        self.lastShowTime = time.time()

    @pyqtSlot()
    def saveCurrentFrame(self):
        filename, formatstr = QFileDialog.getSaveFileName(parent=self, caption="Save Image", directory="untitled.png",
                                           filter="PNG (*.png);;JPEG (*.jpg);;Bitmap (*.bmp);;Portable Bitmap (*.pbm);;Portable Graymap (*.pgm);;Portable Pixmap (*.ppm);;X11 Bitmap (*.xbm);;X11 Pixmap (*.xpm)")
        if(len(filename) == 0):
            return

        if(formatstr == "PNG (*.png)"):
            formatstr = "png"
        elif formatstr == "JPEG (*jpg)":
            formatstr = "jpg"
        elif formatstr == "Bitmap (*.bmp)":
            formatstr = "bmp"
        elif formatstr == "Portable Bitmap (*.pbm)":
            formatstr = "pbm"
        elif formatstr == "Portable Graymap (*.pgm)":
            formatstr = "pgm"
        elif formatstr == "Portable Pixmap (*.ppm)":
            formatstr = "ppm"
        elif formatstr == "X11 Bitmap (*.xbm)":
            formatstr = "xbm"
        elif formatstr == "X11 Pixmap (*.xpm)":
            formatstr = "xpm"
        else:
            self.erroredOut.emit("Invalid format, " + formatstr + ". No image saved.", ErrorPriority.Notice)

        writer = QImageWriter(filename, formatstr.encode('utf-8'))

        if(not writer.canWrite()):
            self.erroredOut.emit("Cannot write image.", ErrorPriority.Notice)

        if(not writer.write(self.currImg)):
            self.erroredOut.emit("Cannot write image. " + writer.errorString(), ErrorPriority.Notice)

_antipattern_ · wrote on 28 Nov 2016, 09:29

QWidget's documentation claims that the QWidget::paintEngine implementation may not always return a valid pointer, which is what I think causes your problem.
One solution might be to override it and provide your own QPaintDevice. On StackOverflow I found a workaround based on QtQuick instead of QtWidget, using a QQuickPaintedItem that you can use inside your QtQuick scenegraph.