Interact with the Real World: OpenXR Scene Understanding

OpenXR Scene Understanding Plugin Setup

Supported Unreal Engine version: 4.27 +

• Enable Plugins:
  • Please enable plugin in Edit > Plugins > Virtual Reality:

• Disable Plugins:
  • The "Steam VR" plugin must be disabled for OpenXR to work.

• • Please disable plugin in Edit > Plugins > Virtual Reality:

• • Project Settings:
    • Please make sure the “ OpenXR Scene Understanding extension ” is enabled, the setting is in Edit > Project Settings > Plugins > Vive OpenXR > Scene Understanding:

Introduction to Blueprint Nodes for OpenXR Scene Understanding

• Used to limit the range that will be scanned
  • These bounding volumes are used to determine which scene components to include in the resulting scene.
    • Set Scene Compute Oriented Box Bound : Set the bounds of the box shape.
    • Set Scene Compute Sphere Bound : Set the bounds of the sphere shape.
    • Set Scene Compute Frustum Bound : Set the bounds of the frustum shape.
    • Clear Scene Compute Bounds: Before the game ends, you need to clear all bounds by calling Clear Scene Compute Bounds .

• Controls the update speed of the scene and will affect the quality of the scene result
  • This function will be trading off speed against the quality of the resulting scene.

• • Set Scene Compute Consistency

• Identify the level of detail of meshes
  • This function will identify the level of detail of visual mesh compute.

• • Set Mesh Compute Lod

Show Scene Understanding scanned mesh in the game

• Step1. Essential Setup
  • • Creating an AR Session:
• The AR-related content needs to be set here because the OpenXR Scene Understanding uses OpenXRARTrackedGeometry to make the scanned meshes appear in the level. Unreal Setting up AR project Tutorial
• Create Data Asset: Content Browser > All > Content > click right mouse button > choose Miscellaneous > Data Asset

• • After choose Data Asset , it will pop out “ Pick Class For Data Asset Instance ” window:

• • • Choose “ ARSessionConfig ” and press “ Select ”.

• • Open ARSessionConfig and enable some settings under Details Panel > AR Settiongs > World Mapping
    • Enable Generate Mesh Data from Tracked Geometry
    • Enable Generate Collision for Mesh Data

• Open Level Blueprint and add Start AR Session and Stop AR Session
  • Remember to set the “ARSessionConfig” as input to Start AR Session.

• Create a Blueprint for Scene Understanding and drag into the level.

• In this tutorial we choose Actor as the parent class.

• In Components panel:
  • Add a Cube Static Mesh Component for Set Scene Compute Oriented Box Bounds .

• Add ARTrackableNotify Component in Components panel.

• • Add a new variable and name it “ spatialMeshes ” with type “ ARTrackedGeometry Array ” in My Blueprint

• Add a function and name it “ StartSpatialMapping ” in My Blueprint In this function, we need to call “ Toggle ARCapture ” function and set input “ On Off ” bool to true and “ Capture Type” to Spatial Mapping.

• • Add a function and name it “ StopSpatialMapping ” in My Blueprint In this function, we need to call “ Toggle ARCapture ” function and set input “ On Off ” bool to false and “ Capture Type” to Spatial Mapping .

• • Event BeginPlay : We will add some functions that affect the quality and area of the scanned mesh.

• • On Add/Remove Tracked Geometry (ARTrackableNotify):
    • These two Events will control the number of mesh currently scanned.

• Event Tick: You can get the current Spatial Meshes and Set Material on them.

• Event BeginPlay: We will need to call Stop Spatial Mapping function and clear the bound you use at the beginning.

Result

Note:

• White Area: meshes representing the real-environment objects.
• Black Area: no objects in the space.

Spawn balls and interact with generated meshes

This sample is currently only supported Unreal Engine version: 5.0+ .

Because the physics system used by the meshes generated is Chaos Physics .

• Create a Blueprint
  • Create a Blueprint for the “Spawn ball” and drag it into the level.
  • You can place as many as you want in the level.

• BP_SpawnBall: Make a sphere static mesh to simulate a physical fall.
  • In Components panel: Add a Sphere Static Mesh Component to pretend the ball in Components panel.

• In Details panel: Enable Simulate Physics and make sure the Collision Presets is “PhysicsActor”

• In My Blueprint panel:
  • Add a variable to hold the origin location of the ball.

• • In Event Graph panel:
    • Set the origin location of the ball when the game starts.

• • • Track the height of the ball in each frame, and reset its position if it is below -600.

Result

Note:

• Red Balls: Virtual Objects.
• White Area: It means that there are scanned meshes .
• Black Area: It means that there are no objects in the space.