Android 14: QCamera In QGraphicsVideoItem Incorrect Orientation

KenAppleby 0

Build platform: android-33; Qt 6.6.0 on Windows.

I have a Pixel 7 phone which recently upgraded to Android 14. My application uses a QGraphicsVideoItem to preview video from the default camera. This has been working more or less okay prior to this upgrade to Android 14. Now the orientation of the video frames in the QGraphicsVideoItem is wrong by 90 degrees after each screen orientation change.

The code below is a small but complete example of this.

The initial view is correctly orientated (though not scaled correctly in the view) but after each screen orientation change from portrait to landscape, or the inverse, the video frames in the QGraphicsItem are 90 degrees out and the QGraphicsItem renders them into a small rectangle as a consequence of this.

The same code running on a Pixel 4a with Android 13 and on an old Sony running Android 8 works as expected, mostly, with the video correctly orientated most, though not all of the time.

What am I missing here please? Is there some error in the way the QGraphicsVideoItem is used in this code?

The code behaves the same when built with Qt 6.5.3.

#pragma once

#include <QMainWindow>
#include <QGraphicsScene>
#include <QCamera>
#include <QGraphicsVideoItem>
#include <QMediaCaptureSession>

class MainWindow : public QMainWindow
{
    Q_OBJECT

public:
    MainWindow(QWidget * parent =nullptr);

    void startCamera();

protected:
    void checkCameraPermissions();

signals:
    void cameraPermissionChecked();

protected:
    QGraphicsScene mScene;

    QCamera * mCamera{ nullptr };

    QGraphicsVideoItem * mGraphicsVideoItem{ nullptr };
    QMediaCaptureSession * mCaptureSession{ nullptr };
};

#include "mainwindow.hpp"
#include <QGraphicsView>
#include <QMainWindow>
#include <QVBoxLayout>

#include <QCameraDevice>
#include <QMediaDevices>
#include <QPermissions>
#include <QTimer>
#include <QCoreApplication>

class MyGraphicsView : public QGraphicsView
{
public:
    MyGraphicsView(QGraphicsVideoItem * videoItem, QWidget * parent);

    void resizeEvent(QResizeEvent * e) override;
    void fitToItems();

    QGraphicsVideoItem * mGraphicsVideoItem{ nullptr };
};

MyGraphicsView::MyGraphicsView(QGraphicsVideoItem * videoItem, QWidget * parent)
    : QGraphicsView{ parent },
      mGraphicsVideoItem{ videoItem }
{
    setVerticalScrollBarPolicy(Qt::ScrollBarAlwaysOff);
    setHorizontalScrollBarPolicy(Qt::ScrollBarAlwaysOff);
}

void MyGraphicsView::resizeEvent(QResizeEvent * e)
{
    QGraphicsView::resizeEvent(e);
    fitToItems();
}

void MyGraphicsView::fitToItems()
{
    fitInView(mGraphicsVideoItem, Qt::KeepAspectRatio);
}

void MainWindow::checkCameraPermissions()
{
    QCameraPermission cameraPermission;
    Qt::PermissionStatus result{ qApp->checkPermission(cameraPermission) };
    if (result == Qt::PermissionStatus::Granted)
    {
        emit cameraPermissionChecked();
    }
    else
    {
        qApp->requestPermission(cameraPermission, this, [&result, this](const QPermission& permission)
        {
            result = permission.status();
            if (result == Qt::PermissionStatus::Granted)
            {
                emit cameraPermissionChecked();
            }
        }
    );
    }
}

MainWindow::MainWindow(QWidget * parent)
    : QMainWindow(parent)
{
    QWidget *centralwidget;
    QVBoxLayout * verticalLayout;
    MyGraphicsView * graphicsView;

    mGraphicsVideoItem = new QGraphicsVideoItem;
    mScene.addItem(mGraphicsVideoItem);

    centralwidget = new QWidget(this);
    verticalLayout = new QVBoxLayout(centralwidget);
    verticalLayout->setSpacing(0);
    verticalLayout->setContentsMargins(0, 0, 0, 0);
    graphicsView = new MyGraphicsView(mGraphicsVideoItem, centralwidget);
    verticalLayout->addWidget(graphicsView);
    setCentralWidget(centralwidget);

    graphicsView->setScene(&mScene);

    connect(this, &MainWindow::cameraPermissionChecked, this, &MainWindow::startCamera);

    QCameraDevice cameraDevice{ QMediaDevices::defaultVideoInput() };
    mCamera = new QCamera{ cameraDevice };
    connect(mCamera, &QCamera::activeChanged, graphicsView, &MyGraphicsView::fitToItems);

    QTimer::singleShot(200, this, [this]() { this->checkCameraPermissions(); });
}

void MainWindow::startCamera()
{
    qDebug() << "starting camera";

    mCaptureSession = new QMediaCaptureSession;
    mCaptureSession->setCamera(mCamera);
    mCaptureSession->setVideoOutput(mGraphicsVideoItem);
    mCamera->start();
}

#include "mainwindow.hpp"

#include <QApplication>

int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
    MainWindow w;
    w.show();
    return a.exec();
}

KenAppleby 0

In the remote chance that anyone is interested in the solution to this ...

Part of my mistake was to assume that the QCamera returned by:

QCameraDevice cameraDevice{ QMediaDevices::defaultVideoInput() };
camera = new QCamera{ cameraDevice };

has a valid QCameraFormat. It doesn't. You have to explictly set the format.

This presumably meant that the QGraphicsVideoItem's QVideoSink didn't know what to do with the QVideoFrames it was receiving, which is fair enough. An error message of some sort would have been helpful.

One of the effects of this is that the video frames were being rendered outside the QGraphicsVideoItem's boundingRect. Giving the camera an explicit QCameraFormat resolves this.

The other issue of the orientation of the video frames in the QGraphicsVideoItem is not solved by this, but the possible solutions are apparent to me now:

If your application is feeding camera video frames into a QGraphicsVideoItem and you want your application to respond to screen orientation changes you have to explicitly control the rotation of the QGraphicsVideoItem in the QGraphicsScene, flipping it by -90, 0, +90, +180 degrees as appropriate.

For my application it is much better to keep things simple and fix the orientation to portrait*.
Then:

set the QGraphicsVideoItem's size to be the transpose of the cameraFormat.resolution().
set the QGraphicsVideoItem's aspect ratio mode to Qt::IgnoreAspectRatio.
use fitInView(mGraphicsRectItem, Qt::KeepAspectRatio); on the view.

The video is then centred on the screen, free of rendering artifacts, with an undistorted aspect ratio, and a good frame rate. It is also possible to meet the original requirement of overlaying the video with either other graphics items or widget children of the view.

*I note that this is what the latest android Camera apps do in android 13 and 14. Instead of the application UI reorganising itself on a main window resize, the app uses the device orientation sensor's information to just flip the individual widgets.

KenAppleby 0

Here's the CMake file:

cmake_minimum_required(VERSION 3.5)

project(GraphicsVideoTest VERSION 0.1 LANGUAGES CXX)

set(CMAKE_AUTOUIC ON)
set(CMAKE_AUTOMOC ON)
set(CMAKE_AUTORCC ON)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

find_package(Qt6 REQUIRED COMPONENTS Core Gui Widgets Multimedia MultimediaWidgets)

set(PROJECT_SOURCES
    main.cpp
    mainwindow.cpp
    mainwindow.hpp
)

qt_add_executable(GraphicsVideoTest ${PROJECT_SOURCES})
target_link_libraries(GraphicsVideoTest PRIVATE Qt6::Core Qt6::Gui Qt6::Widgets Qt6::Multimedia Qt6::MultimediaWidgets)

qt_finalize_executable(GraphicsVideoTest)

KenAppleby 0