Qt3d + Oculus SDK

jahshaka

Thanks Brad for the reply - very helpful :) the Qt 5.7 release is just a few months off so may be better for us to wait for that and then try to get some help with making this happen, since as #kshegunov mentioned the qt3d api should be locked down by then and in better shape for tinkering.

like you said It would really be great if we can get some examples of offscreen rendering etc but doesnt seem to be happening short of paying out mad money so we will just focus on cleaning up our code in the mean time so we can get it out there as well. Its just hard to code in 3d with the VR kit sitting down next to us and not able to use it :) as it really is mind blowing stuff

i also totally agree - VR support should be abstracted so it can work with Oculus and Vive and others - we are trying to get a Vive as well now for testing here as well. We would just love to help push qt3d vr support ahead and start to figure out some of the core issues in the mean time and killer features. Rendering widgets inside 3d space would also be wicked cool I think - like google is doing in tiltbrush

njbrown

@Brad-Austin-Davis said:

So what's needed is a mechanism to do some basic initialization of the Oculus SDK, an means of using the SDK as an input aspect in order to allow the head pose of the HMD to drive the camera position(s) and a means of detecting the rendered frame completion and taking the resulting image in the FBO and pushing it to the Oculus SDK (perhaps something extending QAbstractFrameAdvanceService?).

That is a really good idea, but unfortunately there may be no way for me to set the active QAbstractFrameAdvanceService. VSyncFrameAdvanceService Is created and assigned directly to the renderer:
https://github.com/qtproject/qt3d/blob/5.6/src/render/backend/renderer.cpp#L140
https://github.com/qtproject/qt3d/blob/5.6/src/render/backend/renderer_p.h#L230

In the QRenderAspect, it is being added directly as a service but the service provider isnt being checked for an already set QAbstractFrameAdvanceService implementation:
https://github.com/qtproject/qt3d/blob/5.6/src/render/frontend/qrenderaspect.cpp#L412

Are there any other suggestions to go about doing this without modifying qt3d's source?

Thanks in advance

anda_skoa

@njbrown said:

Are there any other suggestions to go about doing this without modifying qt3d's source?

While that is obviously not an immediate solution, doing that and proposing it through Gerrit could be the best option long term.

That way seems to have been used quite successfully by KDE developers to extend Qt in ways they needed.

Cheers,
_

Dabulla

Hello everyone,

I also wanted to integrate Oculus SDK in qt3d and just started a search if there already have been efforts.

My thoughts so far are:

make it available for everyone on GitHub as a plugin for everyone to use, not part of another bigger project
provide a layer to use OpenVR (Vive) or libOVR (Oculus) directly with the same Qml Objects

Maybe the Qt3D Team is more likely to help, if the result is a standalone VR addition to Qt3D? So they would not have to dig into someone's code.

I wanted to add an Qt3DCore::Entity named "VRManager" or "VRController" which handles

Initialization,
creating an texture swap chain as a Qt3DRender::QRenderTarget
used as a context for all other objects

Other Objects:

In OpenVR I saw there is a list of 3D-Objects (controller, lighthouses, ...) which are tracked. One can iterate through all objects. I wanted to subclass "Qt3DRender::QGeometry" to access these objects in qml (I think libOVR will have/has something similar, at least when touch-controllers are out).

Head pose:

Headset Camera-Entity with a
Read Only: Head Pose, Eye Offsets that can be used as input
Read/Writeable: position, viewCenter, upVector, ... (without applied headPose)

The final Headpose/viewMatrix would not be set by qml, as it is very crucial that this is done just before rendering. The C++ side of VRManager would query libOVR for a predicition of the head pose just before the drawcall. This prediction is then applied to the viewMatrix. Especially this is not a task that can be done in a seperate thread (I think usually all input is handled by the "logic" aspect in another thread).

I was not aware of QAbstractFrameAdvanceService yet... Has anyone tried to subclass the complete QRenderaspect?
Especially when the scene is rendered on the monitor-screen at the same time, it would be best to have a seperate thread (and thus an additional aspect?) for both renderings.

I'd love to see VR in Qt3D and would like to be involved in it's development.

Cheers

Dabulla

It looks like when "Scene3D" is used inside a QQuickView, Qt3D integrates into the rendering cycle of the QQuickView. Maybe this is the solution to call "ovr_CommitTextureSwapChain" and "ovr_SubmitFrame" at the right time.

This would require a special "QQuickView" for VR that synchronizes to the HMD. This would not be able to do VSync to a Monitor.

I'm still not sure how a proper architecture would look like for this case. With one Qml-Context for the scene shown on both devices, the elements for Monitor and HMD would have to update on different speeds.
Two different qml-engines could not share qml-elements (ids), but could synchronize to the two devices and maybe share a mirror-texture of the HMD.

njbrown

@Dabulla you're right, Qt3D uses synchronous rendering when used in a QQuickView, but it did not work in the c++ version the last time I tried. The official version of Qt3D has come out since, so i'll have to try again and tell you guys if it works or not. It's probably the best option for getting vr working right now because you can definitely know when a frame starts and when it ends.

Another hackish option is to use a QFrameAction component to submit and swap the render textures each frame....but that'll only work if QFrameAction runs its callback on the same thread as the renderer. I'll have to investigate this further.

Also, subclassing the QRenderAspect wouldnt do much since the actual rendering takes place in another class called Renderer which isnt publicly accessible. So that option wouldnt work.

njbrown

Has anyone found a creative solution to this problem?

jahshaka

@Dabulla nick - njbrown - is working on a new 3d framework built on top of qt and qopengl that will have native support for the occulus so hit him up, it would be great if you can hep out! Will probably call it Qt3DC (for community edition since Qt3D seems to be enterprise only)

tekojo

@jahshaka
Qt3D isn't enterprise only, it's all part of Qt.
KDAB just have been contributing most to it, and their priorities might not match yours (different kind of target applications bring up different kind of bugs).
So it would probably make more sense to align development efforts in one place and not build another module that for the most part duplicates work. Especially since you would know what to fix, getting patches in is not overly difficult.

kshegunov

Actually @tekojo's wrong (wink), Qt3D desperately needs better docs for the features it has. So it's not even required you to learn its internals, just contributing know-how would greatly improve the module and your understanding of it. I myself have asked on several occasions the lead dev on the mailing how things are done and he's been responsive and understanding. I think you went the wrong way about it if you had expected him/them to give you code/help with your release - that'd require time and time costs are rather steep in our tidy little field. Instead I'd focus on seeking knowledge about the module - how things are done and putting examples/docs with that new knowledge would be very, very much welcome. I'm quite convinced Sean would jump at the occasion of someone helping with the docs (it's one of the reasons why Qt3D is opensource).

jahshaka

@kshegunov we have tried to get help many times and received quotes for paid services on two occasions and zero help whatsoever - not even answers to basic questions. this is totally so un open source its not funny.

Sean has has not answered any of our posts here or on the mailing list thats for sure... and others have the same issue

we have already started on our engine (and it already has oculus support!!) and since its built on QOpenGL it should be easy for others to use and get involved... we hope a community built and supported engine will find a better home and better support

as soon as we have something worthy of publishing we will post a link to our github

kshegunov

@jahshaka said in Qt3d + Oculus SDK:

Sean has has not answered any of our posts here or on the mailing list thats for sure... and others have the same issue

I'm far away from that particular project, much less I want to appear as trying to defend him/kdab. I just want the record to reflect that this is a user forum, devs don't frequent it. As for the mailing list, I see a single question (6th of May) posted, which in all fairness indeed did not get any attention.

Good luck with your engine! I will follow its development with great interest, although I haven't any work with 3D or graphics.
Kind regards.

jahshaka

We are making good progress with our engine - and already have basic VR support

alt text

Dabulla

I just got the first Qt3D Scene on the Oculus with headtracking! It required a small patch in qtbase (qopenglframebufferobject ). I think my patch would not survive a review yet. I use Scene3D with QQuickRendercontrol. It is a proof of concept and I it looks like it is locked at 60Hz at the moment.
I think a bit of engineering is needed to decouple everything from vsync.

? Offline

@Dabulla Awesome!

Dabulla

This is my code (proof of concept):

https://github.com/dabulla/qtbase/tree/virtual-reality
https://github.com/dabulla/qt3d/tree/virtual-reality

both are based on the v5.8.0-beta1 branches.

About:

you may need to change the path of the ovr lib in scene3d-vr.pro
example is located in qt3d/examples/qt3d/scene3d-vr and should work after qtbase was recompiled with my patch. I open example.pro in QtCreator.
The patch in qtbase allows to create an attachment for QOpenGLFramebufferObject by providing a GLuint.
The only classes used for VR at the moment are
- VrRenderer: basically Initialization and swapping buffer to HMD.
- VrCamera: Headtracking, stereo viewports
In WindowMultiThread::setHMDSurface(...) the FrameGraph is searched for a SurfaceSelector and a VrCamera. The SurfaceSelector is reconfigured from outside Qml to choose an offscreen surface. The VrCamera is read to initialize it with ovr and the current HMD. (These are workarounds because I do not yet know about adding aspects. I think there is a nicer solution by using the Entity Component System correctly).
The sample is a combination of qtdeclarative/examples/quick/rendercontrol and the qt3d scene3d example.
- Initialization for VR is done the first time "render" is called.
- Rendering happens into the offscreen texture that comes from ovr using the patched QOpenGLFramebufferObject.
- QuickRenderer additionally calls methods from ovr to "commit" textures to the headset (I think comparable to glSwapBuffers)

Known Issues:

I think rendering is vsynced to the monitor and window at the moment (60Hz).
Oculus only. I just had a closer look at OpenVR after a while and noticed, that it might be able to use a QOpenGLTexture directly! Thus, the patch in qtbase would not be needed for OpenVR. Although I think the Oculus API is a bit ahead of OpenVR at the moment and native support would be desirable.
seg fault sometimes on exit of the application. (maybe the offscreen texture is not cleaned up in the correct order)
No tracked controllers yet, no positional sound, no qml-interface for chaperone/guardian
Im not yet sure how to implement the patch in QOpenGLFramebufferObject cleanly. I want to avoid calling glGet* too often, only to make calls to format() possible. This means QOpenGLFramebufferObject is not constructed correctly. On the other hand side QOpenGLFramebufferObject must be used in order to call QQuickWindow::setRenderTarget(...).

Suggestions are very welcome.

My next step would be to get the example working on the HMDs correct refresh rate and then implement this in a clean way respecting Qt Coding Style. There should be a common vr interface (c++/qml) which can have OpenVR/OculusVR/OSVR as a backend. Then there should be a working demo for the Vive (I heard there is a Vive in one of the KDAB offices :) ).

jahshaka

in our quest for proper VR support, and since we could not afford the high support fees KDAB has out there we have built our own Qt 3D VR engine and it works pretty well...

moved it into its own thread here so please jump in and help out !!!

https://forum.qt.io/topic/74807/irisgl-alpha-code-up