I started adding Virtual Reality support to Qt3D

[Dabulla]

Hi,

I started adding support for OpenVR/Vive and native OculusSDK to Qt3D.

I've been fiddling around with this for a while and finally found a working approach to get an image to the headsets.

What works?

• Stereoscopic 3D Rendering of Qt3D Qml scenes in the Headset.
• Headtracking
• Switching between both VR APIs

What is still in progress?

• Headtracking has high latency (approx. 3 frames)
• Mirroring to Monitor
• Motion controller tracking (coming soon)
• Input
  ... (lots of things imaginable)

Usage

There is QHeadmountedDisplay which does the job of a Qt3DQuickWindow but uses Direct-Mode Rendering. It can be used just like a Window that is placed at the headset and will sync to it's refresh rate. OpenVR or OculusSDK (aka 'SDKs') also handle distortion correctly, so there is no need to do this manually.

The rendered Qt3D Qml scene must use a VrCamera to enable head-tracking and correct stereoscopic rendering. VrCamera configures projection-parameters for the headset by querying the SDK.

I have tested this on windows. It should work on linux as soon as SDKs/drivers support it, maybe it already works with the Vive on Linux.

Setup

Get the folders virtualrealityand vr-window from: https://github.com/dabulla/qt3d-vr/tree/virtual-reality/examples/qt3d
branch: virtual-reality. Note: I will rebase the branch occasionally on qt3d/dev! Do not start work off of it (without telling me).

The implementation is located in qt3d/examples/qt3d/virtualreality (library) and qt3d/examples/vr-window (executable).

It might be possible to use the examples without a custom built Qt3d, as there are no changes outside of the example folder.
However, it uses Qt3D private API for Qt3DRender::QRenderAspectPrivate::renderSynchronous, thus at least the qt3d-private headers are needed.

• Example "vr-window" depends on "virtualreality":
• Change path of SDKs in virtualreality.pro.
• Use preprocessor defines QT3DVR_COMPILE_WITH_OVR and QT3DVR_COMPILE_WITH_OPENVR in qvirtualrealityapi_p.h to enable/disable SDKs. No need to download both SDKs if you only have one!
• use the correct QVirtualRealityApi::Type (Ovr, OpenVR) in vr-window/main.cpp.

Note: The repository also contains scene3d-vr which can be ignored.

Future

I'd really like to proceed with the implementation and look forward to getting some feedback at this point. I read the qt styling guide and started a clean implementation with qt3d/examples/qt3d/virtualreality and wanted to contribute it to qt3d at some point. I'm open for suggestions (especially if you find something that is against the qt design by architecture). I could also need advice with the following problem:

Tackling latency

The pose read from the headset needs approx. 3 frames before it finally ends up as a shader-uniform for rendering. This is 3 frames too long (it is extremely noticable and you get nausea very quickly).

I need a solution to get a QTransform component or Shader uniform updated just before rendering.

Currently it is done like this:

• In the main-thread "getEyeMatrices" is called, this gets the prediction of the head pose for rendering.
• From there the QTransform of the Camera-Entity is set (VrCamera).

I wonder how I can tweak Qt3D to keep this latency down. A tip of a Qt3D expert or architect would be really handy here.

My ideas thus far to tackle latency:

• Introduce QVirtualRealityAspect in backend. Problem: I guess it would not run in the same thread as QRenderAspect. Thus they will need several frames to sync/communicate.
• Introduce a way to set a callback function for a shader uniform. The shader could query a very recent head-pose just before rendering.
• Ensure that "UpdateWorldTransformJob" is executed after setting the transform, but before rendering. Instead of "addDependency()" I needed something like "addThisAsDependencyOf()".
• Don't use synchronous rendering. But this would require me to heavily patch QRenderAspect I guess. At least two not-yet-customizable steps must be altered:
  • Render to a texture owned by the SDK which we only have a GLuint texture-id of (*this is only true for Oculus).
  • Call a custom swapBuffers method.

Hope you enjoy it and don't get sea sick :-)!

[SGaist]

Hi,

Thanks for sharing !

If you would like feedback on your work and get integrated you should create one or more submissions on gerrit. There you'll be able to get reviews from the developers/maintainers of Qt3D.

You can also make your work known on interest mailing list. You'll find there Qt3D developers/maintainers. This forum is more user oriented.

[Michael Levy]

@Dabulla any updates to this? We are interested in contributing to a VR solution for Qt3D (5.14). Would love to chat.

[Dabulla]

@Michael-Levy A contribution would be nice. I plan to update this at the end of the year to newer versions of Qt. No big updates so far.