Skip to content

Add OpenXR-backed VR scene composition and XR controller input#3220

Open
halx99 wants to merge 30 commits into
devfrom
openxr
Open

Add OpenXR-backed VR scene composition and XR controller input#3220
halx99 wants to merge 30 commits into
devfrom
openxr

Conversation

@halx99

@halx99 halx99 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Tested Device and RHIs

Windows: D3D11, D3D12, Vulkan with DP E4 PCVR device

Describe your changes

  • Added optional OpenXR support behind AX_ENABLE_OPENXR, gated by AX_ENABLE_VR and disabled by default.
  • Added the Khronos OpenXR-SDK dependency (release-1.1.60) to the 1kiss dependency profile and third-party CMake integration.
  • Introduced OpenXRDriver, a long-lived runtime backend that owns OpenXR instance, system, session, local space, stereo swapchains, frame timing, action sets, interaction profiles, and controller state.
  • Added VRSceneCompositor, the product VR rendering path backed by OpenXR. It polls XR frame/input state before scheduler update, renders each eye into the runtime swapchain, draws controller rays, submits projection layers, and falls back to normal rendering when the XR session is unavailable.
  • Replaced the old SceneRenderer abstraction with SceneCompositor, now owned by RenderViewCore. Custom scene composition is installed with RenderViewCore::setSceneCompositor().
  • Replaced VRGenericRenderer with VRPreviewSceneCompositor for non-runtime stereo preview and debugging. The preview path keeps side-by-side rendering, distortion meshes, head tracking, render scale, vignette, and VR scissor remapping.
  • Added XR input events through XRInputEvent and XRInputEventListener, covering controller buttons, thumbstick axes, aim poses, grip poses, hand, phase, value, and interaction profile data.
  • Extended pointer input with PointerType::Controller, optional 3D rays, previous rays, and PointerHitResult, allowing XR controller rays to drive existing pointer-based UI and scene interactions.
  • Added VR pointer dispatch helpers to InputSystem, including controller pointer down/move/up/scroll and XR input event dispatch.
  • Updated EventDispatcher to route XR input listeners and store pointer hit results during scene-graph hit testing and pointer capture dispatch.
  • Made Node::onPointerHitTest() public and ray-aware, so custom 2D/3D nodes can participate in controller-ray picking.
  • Updated Widget, ScrollView, GUI scroll views, menus, and terrain picking to support controller rays, hit results, clipping, scroll gestures, and captured pointer streams in VR.
  • Added terrain ray-hit regression coverage and updated cpp-tests to use pointer hit results instead of manually reconstructing terrain rays.
  • Added asymmetric perspective projection support with Mat4::createPerspectiveOffCenter(), used for per-eye OpenXR frusta.
  • Generalized RenderTexturePass so it can target an RHI RenderTarget directly, disable camera override for advanced compositors, and use stable begin/clear/end ordering.
  • Added renderer/RHI support for external runtime-owned textures via ExternalTextureDesc and DriverBase::createTextureFromNativeHandle().
  • Added RenderContext::submitCurrentFrameCommands() and Renderer::submitCurrentFrameCommands() so external runtimes can receive completed rendering work before swapchain release.
  • Added backend support needed by OpenXR swapchains across D3D11, D3D12, Vulkan, OpenGL ES, and shared RHI texture wrapping paths.
  • Added a VulkanInterop abstraction layer. OpenXRVulkanInterop implements the interface and is registered via Application::registerVulkanInterop(). GraphicsCore queries runtime-required Vulkan instance/device extensions and the runtime-selected physical device through this interface, keeping the core RHI free of OpenXR dependencies.
  • Added backend-specific OpenXR graphics bindings for D3D11, D3D12, Vulkan, and Android OpenGL ES. Desktop OpenGL and Windows ANGLE GLES are explicitly treated as unsupported OpenXR graphics backends.
  • Moved scene event polling into the active compositor path so OpenXR frame timing and input are synchronized before game logic updates.
  • Added a VR mode indicator when an active VR compositor is presenting.
  • Renamed DriverContext to GraphicsCore.
  • Updated C++ and Lua templates and VR cpp-tests to use VRPreviewSceneCompositor or VRSceneCompositor instead of VRGenericRenderer.
  • Updated Lua bindings for new base, RHI, and UI APIs.

Issue ticket number and link

Checklist before requesting a review

For each PR

  • Add Copyright if it missed:
    - "Copyright (c) 2019-present Axmol Engine contributors (see AUTHORS.md)."

  • I have performed a self-review of my code.

    Optional:

    • I have checked readme and add important infos to this PR.
    • I have added/adapted some tests too.

For core/new feature PR

  • I have checked readme and add important infos to this PR.
  • I have added thorough tests.

Axmol 3.x ------------------------------------------------------------

For each 3.x PR

  • Check the '#include "axmol.h"' and replace it with the needed headers.

@halx99 halx99 added this to the 3.0.0 milestone Jun 26, 2026
@halx99 halx99 changed the title Integrate OpenXR Integrate OpenXR and HMD support Jun 28, 2026
@halx99 halx99 added enhancement New feature or request experimental VR:HMD labels Jul 2, 2026
@halx99

halx99 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

2 similar comments
@halx99

halx99 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99

halx99 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99

halx99 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99 halx99 force-pushed the openxr branch 2 times, most recently from 73c8376 to 0f7f436 Compare July 3, 2026 15:38
@halx99

halx99 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99 halx99 changed the title Integrate OpenXR and HMD support Add OpenXR-backed VR scene composition and XR controller input Jul 3, 2026
@halx99

halx99 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99

halx99 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99 halx99 marked this pull request as ready for review July 3, 2026 17:53
@halx99

halx99 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99

halx99 commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99

halx99 commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99

halx99 commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99

halx99 commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86a86e064d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread axmol/base/EventDispatcher.cpp
Comment thread axmol/vr/OpenXRDriver.cpp
Comment thread axmol/vr/OpenXRDriver.cpp Outdated
@halx99

halx99 commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

@halx99

halx99 commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

/clang-format

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants