[solved] Using the camera api on the n950

[cpscotti]

I tried looking into http://doc.qt.nokia.com/qtmobility-1.2/qml-camera.html which (afaik) is the official doc to see if we were missing something to no avail.

Soo, We should either fill a bug report to the API or to the docs..

[dmytro.poplavskiy]

I tried this example on device using
qmlviewer -opengl -fullscreen cameratest.qml
and it worked fine. I ssh to user account.

This issue is very likely caused by aegis security system, it's necessary to add file like this

@
<aegis>
<request>
<credential name="GRP::video" />
<credential name="GRP::pulse-access" />
<for path="absolute path to application" />
</request>
</aegis>
@

to debian/appname.aegis
and rebuild/reinstall deb package

[kkrzewniak]

Thank you very much dmytro I figured it was some kind of policy missing and was experimenting with aegis :)
but never found the credential names I needed to include.

Is there perhaps a document describing the credentials needed for certain features.
Thanks

[cpscotti]

Thanks dmytro!
just one thing, the example works now but it segfaults when I swipe it to the background. Do you guys see this too?
I reckon it might be an "expected" behavior since a proper app should put the camera on stand by when inactive..
Any thoughts on this?

[kkrzewniak]

Yeah have the same behavior here but I'm 99% sure this will be solved once I put the camera on stand by (it's required behavior afaik, imagine having your app running in the background with the camera active, preventing other apps form using it).

[cpscotti]

Yep, me too BUT then we have another problem.
How do you put the camera in stand by?
I tried adding:
@
Connections {
target: platformWindow
onActiveChanged: {
if (platformWindow.active) {
console.log("Active")
camera_item.start()
} else {
console.log("Inactive")
camera_item.stop()
}
}
}
@
But it doesn't work at all.

This is the output I get when I swipe it to the background:
@
...
Meego graphics system destroyed
Inactive
ResourceSet::proceedIfImFirst()...allowing only request directly.
ResourceSet::update().... updating...
ResourceEngine(1)::updateResources() - **************** locking....
Converted Resource 0x03 to 0x08
Converted Resource 0x0a to 0x800
Converted Resource 0x0b to 0x1000
All resources as bitmask is 0x1808
ResourceEngine(1) - update 1:5
ResourceSet::proceedIfImFirst()...queuing request 2.
ResourceSet::proceedIfImFirst()...queuing request:Acquire.
Illegal instruction (core dumped)
@

The weird thing is that Camera.start()/Camera.stop() doesn't seem to do anything at all.
Actually, if you remove all your calls to start(), the camera is still loaded normally.
Cool isn't it?

[matrixx]

The reason why aegis credentials are not documented properly yet, is that the final set of credentials hasn't been frozen yet. Proper documentation of which credentials are needed for which functionality, including a complete list of all available credentials will be published later.

[quote author="kkrzewniak" date="1312457295"]Thank you very much dmytro I figured it was some kind of policy missing and was experimenting with aegis :)
but never found the credential names I needed to include.

Is there perhaps a document describing the credentials needed for certain features.
Thanks[/quote]

[cpscotti]

Thanks matrixx for the clarification on the creds.

Do you (or anyone else) have any idea on how to STOP the camera? (so it doesn't crash when windows is inactive)..

[kkrzewniak]

calling stop would suffice the only problem is that
@platformWindow
onActiveChanged@

Is called after I get "Meego graphics system destroyed" and if I remember correctly if I stop the camera before that by hand it's all ok

[dmytro.poplavskiy]

You can try to use the raster graphics system for application with QGLWidget based viewport to ensure camera doesn't loose gl context.

The more correct solution would be to stop() or even better unload() the camera when minimized, as suggested before.

I'm also concerned to see "Meego graphics system destroyed" before "Inactive", it's better to unload the camera before graphics system is destroyed.

[deion]

Hi Dmytro,

I am also trying to use QCamera APIs on N950.
My first issue was on making the aegis manifest file but this is now resolved thanks to the community help.

Now I am having other issues:

1. I am trying to grab frames from the camera. I understand (and you provided this solution before) that for this I need to override QAbstractVideoSurface, re-implement supportedPixelFormats() and present(); I have done that and on my custom QAbstractVideoSurface the method is called with the video frames (QVideoFrame param).

How I used it :

@ camera = new QCamera;
videoSurface = new VideoSurface(); // VideoSurface extends QAbstractVideoSurface
camera->setViewfinder(videoSurface);
camera->setCaptureMode(QCamera::CaptureVideo);
camera->start();@

As I said this works (present is called with desired QVideoFrame s). However during format negotiation in VideoSurface::supportedPixelFormats() the only pixel format available is QVideoFrame::Format_UYVY. This means that in VideoSurface::present I can not use the provided QVideoFrame param to construct a QImage from the raw frame data and using this QImage display the frames on the screen (viewfinder). Should I implement a custom color space conversion from UYVY to RGB to be able to display the frames? I saw in another post of yours that you suggested using this code to inject the custom video surface (http://developer.qt.nokia.com/forums/viewthread/370):

@QVideoRendererControl *control = camera->service()->requestControl<QVideoRendererControl>();
if (control) control->setSurface(yourSurface);@

(and I suppose that instead of
@camera->setViewfinder(videoSurface); @

use

@QCameraViewfinder* viewFinder = new QCameraViewfinder();
camera->setViewFinder(viewFinder);
viewFinder->show();@
which will correctly display a viewfinder)

BUT, in this line
@QVideoRendererControl* control = camera->service()->requestControl<QVideoRendererControl>();@

the returned control is NULL...

So how can I grab QCamera video frames and also present a viewfinder on the screen? The only solution is through colorspace conversion? This is time consuming ...

2. Another issue is how to set the desired video resolution? I used a QMediaRecoder and set the desired settings on this object to force desired video resolution on QCamera(even though I am not interested in capturing video on a file, I am only interested of grabbing video frames at certain resolution and FPS):

@ camera = new QCamera;
videoSurface = new VideoSurface(); // VideoSurface extends QAbstractVideoSurface
camera->setViewfinder(videoSurface);
camera->setCaptureMode(QCamera::CaptureVideo);

mediaRecorder = new QMediaRecorder(camera);
QVideoEncoderSettings videoSettings = mediaRecorder->videoSettings();
videoSettings.setResolution(QSize(320, 240));
mediaRecorder->setEncodingSettings(mediaRecorder->audioSettings(),videoSettings);

camera->start();@

Thanks & regards,
Ionut Dediu

[dmytro.poplavskiy]

Hi Ionut,

you spotted two areas I'd like to improve in next versions on QCamera API:
to have access to video frames simultaneously with efficient rendering (this is also related by lesser extent to QMediaPlayer) and viewfinder resolution settings (and probably framerate, not so sure about this).

[quote author="deion" date="1314176848"]Hi Dmytro,

I am also trying to use QCamera APIs on N950.
My first issue was on making the aegis manifest file but this is now resolved thanks to the community help.

Now I am having other issues:

1. I am trying to grab frames from the camera. I understand (and you provided this solution before) that for this I need to override QAbstractVideoSurface, re-implement supportedPixelFormats() and present(); I have done that and on my custom QAbstractVideoSurface the method is called with the video frames (QVideoFrame param).

How I used it :

@ camera = new QCamera;
videoSurface = new VideoSurface(); // VideoSurface extends QAbstractVideoSurface
camera->setViewfinder(videoSurface);
camera->setCaptureMode(QCamera::CaptureVideo);
camera->start();@

As I said this works (present is called with desired QVideoFrame s). However during format negotiation in VideoSurface::supportedPixelFormats() the only pixel format available is QVideoFrame::Format_UYVY. This means that in VideoSurface::present I can not use the provided QVideoFrame param to construct a QImage from the raw frame data and using this QImage display the frames on the screen (viewfinder). Should I implement a custom color space conversion from UYVY to RGB to be able to display the frames?
[/quote]

Colorspace transformation is disabled in camerabin pipeline by default for performance reasons, it's possible to enable it with

QCameraControl *control = camera->service()->requestControl<QCameraControl>();
control->setProperty("viewfinderColorSpaceConversion", true);

This flag is gstreamer backend specific (it should be documented).
You may also try to report a limited supported set of RGB formats, gst colorspace convertor on N950 can convert only to a few RGB formats efficiently.

You can also have a look at qgraphicsvideoitem_maemo5.cpp for a very fast UYVY to RGB16 transformation, if license is suitable for you.

I'm thinking about video frames probe like API to have read only access to video frames in parallel with efficient rendering, and video filter like API to modify video frame before rendering, but I can't promise when/if it's done.
Any feedback/ideas in this area are very welcome.

[quote]
I saw in another post of yours that you suggested using this code to inject the custom video surface (http://developer.qt.nokia.com/forums/viewthread/370):

@QVideoRendererControl *control = camera->service()->requestControl<QVideoRendererControl>();
if (control) control->setSurface(yourSurface);@

(and I suppose that instead of
@camera->setViewfinder(videoSurface); @

use

@QCameraViewfinder* viewFinder = new QCameraViewfinder();
camera->setViewFinder(viewFinder);
viewFinder->show();@
which will correctly display a viewfinder)

BUT, in this line
@QVideoRendererControl* control = camera->service()->requestControl<QVideoRendererControl>();@

the returned control is NULL...

[/quote]

Regarding "camera->setViewfinder(videoSurface);" vs request control, those methods should work the same if only one output is used (setViewfinder() detaches previous output) and camera->setViewfinder() is simpler and more preferred.

QMediaService API allows backend to have multiple video outputs attached, but current gstreamer backend doesn't support this, that's why requestControl<QVideoRendererControl>() returns NULL.

[quote]

So how can I grab QCamera video frames and also present a viewfinder on the screen? The only solution is through colorspace conversion? This is time consuming ...

2. Another issue is how to set the desired video resolution? I used a QMediaRecoder and set the desired settings on this object to force desired video resolution on QCamera(even though I am not interested in capturing video on a file, I am only interested of grabbing video frames at certain resolution and FPS):

@ camera = new QCamera;
videoSurface = new VideoSurface(); // VideoSurface extends QAbstractVideoSurface
camera->setViewfinder(videoSurface);
camera->setCaptureMode(QCamera::CaptureVideo);

mediaRecorder = new QMediaRecorder(camera);
QVideoEncoderSettings videoSettings = mediaRecorder->videoSettings();
videoSettings.setResolution(QSize(320, 240));
mediaRecorder->setEncodingSettings(mediaRecorder->audioSettings(),videoSettings);

camera->start();@

[/quote]

Unfortunately the current QCamera API doesn't provide a viewfinder resolution control,
using QMediaRecorder is a valid workaround. It should not have any negative performance impact.

[deion]

Hi Dmytro,

Thanks a lot for your answers, they were really helpful.
I wanted to post a longer reply to highlight more issues regarding the differences between Symbian and Meego regarding the camera capabilities; I will do this later, maybe it will be useful to you to get more feedback on the usage scenarios for the QCamera APIs.
Right now I wanted to ask you about the method
@bool QAbstractVideoSurface::present ( const QVideoFrame & frame )@
What is the lifetime of the frame param? I can not cache it and use it later after returning from the present() method, is this right?
If so, how can I safely make a copy of the frame param into another QVideoFrame object so that I can use it later? I'm thinking of adding the frames into a queue/list and later process these separately from another thread. How can I copy a QVideoFrame?

Thanks & regards,
Ionut

[dmytro.poplavskiy]

Hi Ionut,

[quote author="deion" date="1314345892"]
Right now I wanted to ask you about the method
@bool QAbstractVideoSurface::present ( const QVideoFrame & frame )@
What is the lifetime of the frame param? I can not cache it and use it later after returning from the present() method, is this right?
If so, how can I safely make a copy of the frame param into another QVideoFrame object so that I can use it later? I'm thinking of adding the frames into a queue/list and later process these separately from another thread. How can I copy a QVideoFrame?
[/quote]

It's safe to keep the video frame, it's reference counted and once the video frame is not used any more, the internal video buffer is returned back to media framework. This allows to avoid unnecessary memory copies.

The most common QAbstractVideoSurface::present() implementation includes saving the video frame and calling update() to initialize redrawing of video item/widget. The frame is kept by video item until the next frame arrives.

It's necessary to release the video frame with m_frame = QVideoFrame() when video surface is stopped.

[weiwei2]

Hi

i not too sure how to generate the file as mentioned by "Harmattaneveloper Library/Developing for Harmattan/Harmattan security/Security guide/Aegis manifest file generation tool":http://www.developer.nokia.com/Community/Wiki/Harmattan:Developer_Library/Developing_for_Harmattan/Harmattan_security/Security_guide/Aegis_manifest_file_generation_tool and in the "documentation":http://harmattan-dev.nokia.com/docs/library/html/guide/html/Developer_Library_Developing_for_Harmattan_Harmattan_security_Security_guide_Aegis_manifest_file_generation_tool.html. is there a way to do it in Qt SDK and how?

i'm originally in this "thread":http://www.developer.nokia.com/Community/Discussion/showthread.php?228456-how-to-utilize-some-hardwares-on-N950 but thought asking here for the aegis settings might be more relevant

[Dynasty1215]

Hi, I have the same problem likes yours
1.
If i used
surface = new myVideoSurface();
camera->setViewfinder(surface);

QVideoRendererControl control = qobject_cast<QVideoRendererControl>(camera_->service()->requestControl("com.nokia.Qt.QVideoRendererControl/1.0"));

the control is not null, call control->setSurface(surface);
then the framework can call the supportedPixelFormats start function
But that not display anything on the screen; and there are errors:
Failed to start video surface
Internal data flow error

2.If i used
viewfinder = new QCameraViewfinder();
viewfinder->show();
camera_->setViewfinder(viewfinder);
the screen display the camera's graphics
But then call
QVideoRendererControl control = qobject_cast<QVideoRendererControl>(camera_->service()->requestControl("com.nokia.Qt.QVideoRendererControl/1.0"));
the control is null.
that the framework not call the supportedPixelFormats start function.

Hi,@deion, did you sovled it? and how did you solved it?
Hi,@dmytro, give me some suggestion?
Have othres have the same problem? plz give helps.
Thx.