Integrate Hand Tracking Data Into Your Hand Model

OpenXR Hand Tracking Plugin Setup

Supported Unreal Engine version: 4.26 +

• Enable Plugins:
  • Please enable “ OpenXR ” and “ OpenXRHandTracking ” plugins in Edit > Plugins > Virtual Reality.

• Disable Plugins:
  • The " Steam VR " plugin must be disabled for OpenXR to work, please disable plugin in Edit > Plugins > Virtual Reality.

Get OpenXR Hand Tracking Data

Getting Detection Result.

Detected left and right hand results are available from blueprint function “ GetMotionControllerData ”, and select left or right Hand.

Render Hands using debug lines

Use XRMotionControllerData to draw the debug lines/spheres and form hands.

• Enable Plugin:
  • Please enable “XRVisualization” plugin in Edit > Plugins > Mixed Reality.

• • Create a Blueprint and choose Actor as the parent class:
    • In the Event Graph, we will call “Get Motion Controller Data” to get the hand tracking detection result.
    • After getting the hand tracking detection results, we will call the “ RenderMotionController ” node from the XRVisualization plugin to draw the debug line/sphere hands.

Result

Feed OpenXR HandTracking Data to Skeletal Mesh

Use XRMotionControllerData to update Skeletal Mesh hands transform.

Note: The hand skeleton models used in this Tutorial is the models in the ViveOpenXR sample.

• • Import hand skeleton models.
  • Create a Blueprint for HandTracking (The following is called BP_RiggedHand ).

• • In this tutorial we choose Actor as the parent class.
  • Add Poseable Mesh Component for the hand skeleton mesh in Components panel.

• • • Set Hand Skeletal Mesh in Details panel > Mesh > Skeletal Mesh.

• • Add a new variable and name it “ Hand ” with type “ EControllerHand (enum) ” in My Blueprint We will use this variable to set which hand the current Actor is in the Level.

Go back to Level Editor default Interface, we will drag two “BP_RiggedHand” into Level (for both hands).

• In Outliner panel > click BP_RiggedHand > Details panel > Hand Settings > Hand (Choose Right/Left hand).

Remember to change the Hand Skeletal Mesh for each hand.

• Add a new variable and name it “ JointNames ” with type “ Name Array ” in My Blueprint Joint names are the bones name from the hand skeletal mesh. Later we will use this variable to update each hand bones transform.

• Add a new variable and name it “ HandKeyPositions ” with type “ Vector Array ” in My Blueprint panel . We will use this array to hold the Hand Tracking Positions data in each Tick.

• Add a new variable and name it “ HandKeyRotations ” with type “ Quat Array ” in My Blueprint. We will use this array to hold the Hand Tracking Rotations data in each Tick.

• Event Tick : Get HandKeyPositions/ HandKeyRotations from “ Get Motion Controller Data” . We will use the “ Hand ” variable mentioned in the previous step to determine which hand data is the hand data to be obtained now. Set the hand tracking data obtained from this function to “ HandKeyPosition ” and “ HandKeyRotations ”.

• Event Tick : Get Is Valid from “ Get Motion Controller Data” . We can use one of the output “ Valid ” in XRMotionControllerData to determine whether to show/hide the hand model.

• Add a function and name it “ UpdateHandJointsTransfrom ” in My Blueprint

• • Add some local variables in UpdateHandJointsTransform :

• Call “ ForEach EHandKeypoint ” and set the output ( Enum ) to “ TmpEnumValue ”, we will use it to check IsWrist and get the current hand bone location and rotation , after getting the hand position/ rotation and set them to “TmpJointPose” and “TmpJointRot” .

• If current bone is Wrist, we will use TmpJointPose to update this actor world location.

• Use TmpJointRot to update the current hand bone component space rotation , in this sample we need to add some offsets to correct the Axis of the hand skeleton mesh.

• Note: Since the Hand Tracking Engine does not currently support metacarpal bones rotation, you need to ignore them when updating the rotation of each bone.

• Drag the UpdateHandJointsTransform function into the Event Graph and connect it after the Set Visiblity function, this action means that the game will call the UpdateHandJointsTransform function every tick (If the hand tracking data is valid.)

• Complete function flow in Event Graph

Result