Qt3d + Oculus SDK
-
Hello everyone,
I also wanted to integrate Oculus SDK in qt3d and just started a search if there already have been efforts.
My thoughts so far are:
- make it available for everyone on GitHub as a plugin for everyone to use, not part of another bigger project
- provide a layer to use OpenVR (Vive) or libOVR (Oculus) directly with the same Qml Objects
Maybe the Qt3D Team is more likely to help, if the result is a standalone VR addition to Qt3D? So they would not have to dig into someone's code.
- I wanted to add an Qt3DCore::Entity named "VRManager" or "VRController" which handles
- Initialization,
- creating an texture swap chain as a Qt3DRender::QRenderTarget
- used as a context for all other objects
- Other Objects:
- In OpenVR I saw there is a list of 3D-Objects (controller, lighthouses, ...) which are tracked. One can iterate through all objects. I wanted to subclass "Qt3DRender::QGeometry" to access these objects in qml (I think libOVR will have/has something similar, at least when touch-controllers are out).
- Head pose:
- Headset Camera-Entity with a
- Read Only: Head Pose, Eye Offsets that can be used as input
- Read/Writeable: position, viewCenter, upVector, ... (without applied headPose)
The final Headpose/viewMatrix would not be set by qml, as it is very crucial that this is done just before rendering. The C++ side of VRManager would query libOVR for a predicition of the head pose just before the drawcall. This prediction is then applied to the viewMatrix. Especially this is not a task that can be done in a seperate thread (I think usually all input is handled by the "logic" aspect in another thread).
I was not aware of QAbstractFrameAdvanceService yet... Has anyone tried to subclass the complete QRenderaspect?
Especially when the scene is rendered on the monitor-screen at the same time, it would be best to have a seperate thread (and thus an additional aspect?) for both renderings.I'd love to see VR in Qt3D and would like to be involved in it's development.
Cheers
-
It looks like when "Scene3D" is used inside a QQuickView, Qt3D integrates into the rendering cycle of the QQuickView. Maybe this is the solution to call "ovr_CommitTextureSwapChain" and "ovr_SubmitFrame" at the right time.
This would require a special "QQuickView" for VR that synchronizes to the HMD. This would not be able to do VSync to a Monitor.
I'm still not sure how a proper architecture would look like for this case. With one Qml-Context for the scene shown on both devices, the elements for Monitor and HMD would have to update on different speeds.
Two different qml-engines could not share qml-elements (ids), but could synchronize to the two devices and maybe share a mirror-texture of the HMD. -
@Dabulla you're right, Qt3D uses synchronous rendering when used in a QQuickView, but it did not work in the c++ version the last time I tried. The official version of Qt3D has come out since, so i'll have to try again and tell you guys if it works or not. It's probably the best option for getting vr working right now because you can definitely know when a frame starts and when it ends.
Another hackish option is to use a QFrameAction component to submit and swap the render textures each frame....but that'll only work if QFrameAction runs its callback on the same thread as the renderer. I'll have to investigate this further.
Also, subclassing the QRenderAspect wouldnt do much since the actual rendering takes place in another class called Renderer which isnt publicly accessible. So that option wouldnt work.
-
It looks like when "Scene3D" is used inside a QQuickView, Qt3D integrates into the rendering cycle of the QQuickView. Maybe this is the solution to call "ovr_CommitTextureSwapChain" and "ovr_SubmitFrame" at the right time.
This would require a special "QQuickView" for VR that synchronizes to the HMD. This would not be able to do VSync to a Monitor.
I'm still not sure how a proper architecture would look like for this case. With one Qml-Context for the scene shown on both devices, the elements for Monitor and HMD would have to update on different speeds.
Two different qml-engines could not share qml-elements (ids), but could synchronize to the two devices and maybe share a mirror-texture of the HMD.@Dabulla nick - njbrown - is working on a new 3d framework built on top of qt and qopengl that will have native support for the occulus so hit him up, it would be great if you can hep out! Will probably call it Qt3DC (for community edition since Qt3D seems to be enterprise only)
-
@Dabulla nick - njbrown - is working on a new 3d framework built on top of qt and qopengl that will have native support for the occulus so hit him up, it would be great if you can hep out! Will probably call it Qt3DC (for community edition since Qt3D seems to be enterprise only)
@jahshaka
Qt3D isn't enterprise only, it's all part of Qt.
KDAB just have been contributing most to it, and their priorities might not match yours (different kind of target applications bring up different kind of bugs).
So it would probably make more sense to align development efforts in one place and not build another module that for the most part duplicates work. Especially since you would know what to fix, getting patches in is not overly difficult. -
@Dabulla nick - njbrown - is working on a new 3d framework built on top of qt and qopengl that will have native support for the occulus so hit him up, it would be great if you can hep out! Will probably call it Qt3DC (for community edition since Qt3D seems to be enterprise only)
Actually @tekojo's wrong (wink), Qt3D desperately needs better docs for the features it has. So it's not even required you to learn its internals, just contributing know-how would greatly improve the module and your understanding of it. I myself have asked on several occasions the lead dev on the mailing how things are done and he's been responsive and understanding. I think you went the wrong way about it if you had expected him/them to give you code/help with your release - that'd require time and time costs are rather steep in our tidy little field. Instead I'd focus on seeking knowledge about the module - how things are done and putting examples/docs with that new knowledge would be very, very much welcome. I'm quite convinced Sean would jump at the occasion of someone helping with the docs (it's one of the reasons why Qt3D is opensource).
Read and abide by the Qt Code of Conduct
-
Actually @tekojo's wrong (wink), Qt3D desperately needs better docs for the features it has. So it's not even required you to learn its internals, just contributing know-how would greatly improve the module and your understanding of it. I myself have asked on several occasions the lead dev on the mailing how things are done and he's been responsive and understanding. I think you went the wrong way about it if you had expected him/them to give you code/help with your release - that'd require time and time costs are rather steep in our tidy little field. Instead I'd focus on seeking knowledge about the module - how things are done and putting examples/docs with that new knowledge would be very, very much welcome. I'm quite convinced Sean would jump at the occasion of someone helping with the docs (it's one of the reasons why Qt3D is opensource).
@kshegunov we have tried to get help many times and received quotes for paid services on two occasions and zero help whatsoever - not even answers to basic questions. this is totally so un open source its not funny.
Sean has has not answered any of our posts here or on the mailing list thats for sure... and others have the same issue
we have already started on our engine (and it already has oculus support!!) and since its built on QOpenGL it should be easy for others to use and get involved... we hope a community built and supported engine will find a better home and better support
as soon as we have something worthy of publishing we will post a link to our github
-
@kshegunov we have tried to get help many times and received quotes for paid services on two occasions and zero help whatsoever - not even answers to basic questions. this is totally so un open source its not funny.
Sean has has not answered any of our posts here or on the mailing list thats for sure... and others have the same issue
we have already started on our engine (and it already has oculus support!!) and since its built on QOpenGL it should be easy for others to use and get involved... we hope a community built and supported engine will find a better home and better support
as soon as we have something worthy of publishing we will post a link to our github
@jahshaka said in Qt3d + Oculus SDK:
Sean has has not answered any of our posts here or on the mailing list thats for sure... and others have the same issue
I'm far away from that particular project, much less I want to appear as trying to defend him/kdab. I just want the record to reflect that this is a user forum, devs don't frequent it. As for the mailing list, I see a single question (6th of May) posted, which in all fairness indeed did not get any attention.
Good luck with your engine! I will follow its development with great interest, although I haven't any work with 3D or graphics.
Kind regards. -
I just got the first Qt3D Scene on the Oculus with headtracking! It required a small patch in qtbase (qopenglframebufferobject ). I think my patch would not survive a review yet. I use Scene3D with QQuickRendercontrol. It is a proof of concept and I it looks like it is locked at 60Hz at the moment.
I think a bit of engineering is needed to decouple everything from vsync. -
I just got the first Qt3D Scene on the Oculus with headtracking! It required a small patch in qtbase (qopenglframebufferobject ). I think my patch would not survive a review yet. I use Scene3D with QQuickRendercontrol. It is a proof of concept and I it looks like it is locked at 60Hz at the moment.
I think a bit of engineering is needed to decouple everything from vsync.@Dabulla Awesome!
-
This is my code (proof of concept):
https://github.com/dabulla/qtbase/tree/virtual-reality
https://github.com/dabulla/qt3d/tree/virtual-realityboth are based on the v5.8.0-beta1 branches.
About:
- you may need to change the path of the ovr lib in
scene3d-vr.pro - example is located in qt3d/examples/qt3d/scene3d-vr and should work after qtbase was recompiled with my patch. I open
example.proin QtCreator. - The patch in qtbase allows to create an attachment for QOpenGLFramebufferObject by providing a GLuint.
- The only classes used for VR at the moment are
- VrRenderer: basically Initialization and swapping buffer to HMD.
- VrCamera: Headtracking, stereo viewports
- In
WindowMultiThread::setHMDSurface(...)the FrameGraph is searched for a SurfaceSelector and a VrCamera. The SurfaceSelector is reconfigured from outside Qml to choose an offscreen surface. The VrCamera is read to initialize it with ovr and the current HMD. (These are workarounds because I do not yet know about adding aspects. I think there is a nicer solution by using the Entity Component System correctly). - The sample is a combination of qtdeclarative/examples/quick/rendercontrol and the qt3d scene3d example.
- Initialization for VR is done the first time "render" is called.
- Rendering happens into the offscreen texture that comes from ovr using the patched QOpenGLFramebufferObject.
- QuickRenderer additionally calls methods from ovr to "commit" textures to the headset (I think comparable to glSwapBuffers)
Known Issues:
- I think rendering is vsynced to the monitor and window at the moment (60Hz).
- Oculus only. I just had a closer look at OpenVR after a while and noticed, that it might be able to use a QOpenGLTexture directly! Thus, the patch in qtbase would not be needed for OpenVR. Although I think the Oculus API is a bit ahead of OpenVR at the moment and native support would be desirable.
- seg fault sometimes on exit of the application. (maybe the offscreen texture is not cleaned up in the correct order)
- No tracked controllers yet, no positional sound, no qml-interface for chaperone/guardian
- Im not yet sure how to implement the patch in QOpenGLFramebufferObject cleanly. I want to avoid calling glGet* too often, only to make calls to format() possible. This means QOpenGLFramebufferObject is not constructed correctly. On the other hand side QOpenGLFramebufferObject must be used in order to call QQuickWindow::setRenderTarget(...).
Suggestions are very welcome.
My next step would be to get the example working on the HMDs correct refresh rate and then implement this in a clean way respecting Qt Coding Style. There should be a common vr interface (c++/qml) which can have OpenVR/OculusVR/OSVR as a backend. Then there should be a working demo for the Vive (I heard there is a Vive in one of the KDAB offices :) ).
- you may need to change the path of the ovr lib in
-
in our quest for proper VR support, and since we could not afford the high support fees KDAB has out there we have built our own Qt 3D VR engine and it works pretty well...
moved it into its own thread here so please jump in and help out !!!
