Android: QAbstractVideoSurface not triggered by QMediaPlayer to grab frames from video

and.fo

Hi there,

I'm trying to grab videos from a video file using a QMediaPlayer in combination with a derivation of QAbstractVideoSurface, in order to get single frames in the present-method.

It works well on windows, but on Android the methods in the derived QAbstractVideoSurface are not called. No error in output visible and no error message in the error-method. The isActive method returns false, so it seems that the methods in the the derived classes are not called.

The start-methods and all other methods are not called. The same code works on windows. Frames are grabbed.

Any ideas? is this an bug in QT 5.2 -> I'm using the beta version.

z.emb

I have the same problem with QCamera instead of QMediaPlayer. The Camera switches into ActiveState, even the flashlight works, but no calls to the derived VideoSurface, no start(), no present().

Instead I get "E/Camera : Error 1001" as error.

The exactly same code works on the Desktop. Using Qt5.2

MartynW

I too have much the same problem with the released Qt 5.2 and QCamera (I extended the example http://qt-project.org/doc/qt-4.8/multimedia-videowidget.html). It works on Windows but not Android (Nexus 7).

Weird thing is for Windows camera->isCaptureModeSupported(QCamera::CaptureViewfinder) returns false, but QAbstractVideoSurface::present() gets called, whereas it returns true for Android but present() does not get called as it should.

For Android start() gets called and then present() gets called just the once and then nothing. From start() I see format.pixelFormat() is Format_BGR32 which is also the format in present() but the frame.startTime() is set to -1, which doesn't seem obviously right.

Sounds like it may be a bug.

MartynW

Yep, its been reported in https://bugreports.qt-project.org/browse/QTBUG-34888 and looks like we may get a fix in 5.2.1 according to https://bugreports.qt-project.org/browse/QTBUG-35416.

Meanwhile, weirdly, simply referencing the frames handle() method within your present() method seems to kick the triggering process past the first frame.
@
qDebug() << frame.handleType();
QVariant vHandle = frame.handle();
qDebug() << "Frame: " << frame.startTime() << " format: " << frame.pixelFormat()
<< " valid?: " << frame.isValid() << " size " << frame.width()
<< " x " << frame.height();
@

gives:
@
D/Qt ( 7020): ConsumerVideoSurface.cpp:145 (virtual bool ConsumerVideoSurface::present(const QVideoFrame&)): UserHandle(1001)
D/Qt ( 7020): ConsumerVideoSurface.cpp:165 (virtual bool ConsumerVideoSurface::present(const QVideoFrame&)): Frame: -1 format: Format_BGR32 valid?: true size 960 x 720
@

johnc

I am getting a similar issue.

Putting in the frame.handle() call in my present() method kick started it, but nothing is painting.

I think I must be hitting the raw frame issue as outlined in one of the bug reports.

In the paint function it fails on the frame.map() function. It seems that on Android it is unable to copy the raw frame data over to accessible memory.

Does anyone know how to get around this issue? We need raw access so we can paint to a specific control, and also run a Qr reader over the frames.

The code runs fine on iOS so the error is definitely just Android specific.

MartynW

Supposedly you will be able to get the frame using QVideoProbe in 5.2.1 to be released beginning of Feb time (?) or the proper fix in 5.3 end of April time. Meantime it doesn't look possible (see http://qt-project.org/forums/viewthread/36973/) . Unless someone knows differently!

MartynW

Yes, QVideoProbe works fine in 5.2.1 (don't forget to call frame.map()) and returns image frames in NV21 format, altho the orientation does not change and is at 90 degrees when in portrait mode. It gives consecutive frames even when you don't call frame.handle() in the VideoSurface (which is best as the callbacks don't return anything useful).

and.fo

Grabbing frames is still not working for me on Android.

Android version is 4.4.3. Qt version 5.2.1 as Qt 5.3. crashes for another reason.

Either with a custom QAbstractVideoSurface or the QVideoProbe there is no callback triggered to "present" or "videoFrameProbed".

In case of QVideoProbe the following function returns false on Android and Windows7.

@
probe.setSource(&m_player)}
@

How did you manage it? Is the frame grabbing working on Android?

MartynW

Yes, it works fine for Windows an Android. I vaguely remember setSource returning false at one time, I think I just got the camera name wrong. I show a snippet of what I have below in the hope it may help.

BTW, I think the 'proper' fix never made it into 5.3.

@ QVideoProbe *m_pProbe;

// Search for system camera devices: We pick the first if any specified one is not found
foreach (const QByteArray &deviceName, QCamera::availableDevices())
{
    QString description = m_pCamera->deviceDescription(deviceName);
    if (cameraDevice.isEmpty())
        cameraDevice = deviceName;

    if (description.compare(strCameraName, Qt::CaseInsensitive) == 0)
        cameraDevice = deviceName;

    qDebug() << "Camera: " << description; 
}

// Example device on Windows: "@device:pnp:\\?\usb#vid_0c45&pid_6409&mi_00#7&373d688d&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global"
// Example device on Nexus 7 2012: "front", description "Front-facing camera"

if (cameraDevice.isEmpty())
    m_pCamera = new QCamera;
else
    m_pCamera = new QCamera(cameraDevice);

m_pConsumerVideoSurface = new ConsumerVideoSurface(this);
m_pCamera->setViewfinder(m_pConsumerVideoSurface);
connect(m_pConsumerVideoSurface, SIGNAL(signalOnFrame(QImage)), this , SLOT(slotOnFrame(QImage)), Qt::DirectConnection);
if (bIsVideoProbeRequired)
{
    m_pProbe = new QVideoProbe(this);
    m_pConsumerVideoSurface->SuppressVideo(true);
}


m_pProbe = new QVideoProbe(this);
if (m_pProbe != NULL)
{
    m_pProbe->setSource(m_pCamera); // Returns true, hopefully.
    connect(m_pProbe, SIGNAL(videoFrameProbed(QVideoFrame)), this, SLOT(slotOnProbeFrame(QVideoFrame)), Qt::DirectConnection);
}
m_pCamera->start();

@

and.fo

Thanks a lot. I will have a try :-)

But in may application I'm not grabbing frames from the camera but from the mediaplayer. I'm working on frames from a video playback file.

Maybe this is not working on Android? Do you have any experience with that?

MartynW

Sorry, no, I was only interested in the camera, but I thought QVideoProbe was meant to be working generally so I would expect you to be ok.

and.fo

Thanks again!

I found this post, mentioning QVideoProbe is working only for the camera on Android, not for the MediaPlayer.

http://lists.qt-project.org/pipermail/interest/2014-February/011125.html

MartynW

It seems to be a bit of a work in progress with changes proposed then delayed - see "https://bugreports.qt-project.org/browse/QTBUG-35416":https://bugreports.qt-project.org/browse/QTBUG-35416.

johnc

Is there any further update to this or solution?

The bug reports all seem to say to use QVideoProbe instead, but I can't get that to work on Android ( 4.1.2 ) and QT 5.4.
If I leave out setViewFinder I get QCamera::InvalidRequestError

Putting setViewFinder in removes all errors, the camera changes state from Loading, to Loaded, to Active.
But then after this no frames are captured.

I have seen a number of examples like this one http://www.programering.com/a/MTN3IjMwATQ.html
But all seem to have the same result, no frames.