Palm UI Using PalmMenu in MetaQuest

0. Contents of This Article

This article introduces how to implement PalmMenu that displays UI on the palm and operates it with fingers of the other hand in Meta Quest. PalmMenu can achieve more natural interaction that appeals to the sense of touch compared to floating UI in mid-air, as you actually trace or tap the palm to operate it, making it a very useful feature.
You can also check the operation in the 07-PalmMenu or 07-PalmMenu-AR scene of the sample published on GitHub.

*This video is the AR version but supports both AR/VR.

This article proceeds based on VR or AR scenes created in the following articles.

[VR Version]

Displaying Objects in MetaQuest

[AR Version]

AR Display Using MetaQuest Passthrough

1. Duplicating the Scene

While it's possible to edit the scene created in the previous article, to try PalmMenu without destroying the existing scene, I'll introduce how to duplicate the scene. Skip this if unnecessary.

• Open a VR or AR compatible scene
• Click File -> Save As... to save the current scene with a new name
  *This article uses PalmMenu
• Confirm that the scene name displayed in Hierarchy is PalmMenu

2. Scene Preparation

• Delete the Cube used in the duplicate source as it's unnecessary
• Enter PalmMenu in the Project tab search area
• Select All from Search: directly under the search area
• Two PalmMenus will be found, but select the one stored in the following directory
  Packages/com.meta.xr.sdk.interaction.ovr/Runtime/Sample/Prefabs/
  *Folder name is displayed at the bottom of the Project tab

• Drag and drop PalmMenu prefab into Hierarchy
• Right-click in the empty space of Hierarchy
• Click UI -> EventSystem
• Confirm that EventSystem has been added to Hierarchy
• Select EventSystem and display Inspector
• Click Add Component at the bottom of Inspector
• Search for Pointable Canvas Module and add
• Right-click PalmMenu object
• To make editing easier later, click Prefab -> Unpack Completely

3. PalmMenu Settings

Hand Ref Settings

PalmMenu has two Hand Refs attached to link with hand tracking status, but Hand is unset with None. Associate these with left and right hand information.

[Associate first Hand Ref with left hand]

• Open OVRInteractionComprehensive child element of OVRCameraRig object
• Confirm that there's LeftInteractions as its child element

• Click PalmMenu
• Drag and drop LeftInteractions to Hand in the first Hand Ref

[Associate second Hand Ref with right hand]

• Open OVRInteractionComprehensive child element of OVRCameraRig object
• Confirm that there's RightInteractions as its child element
• Click PalmMenu
• Drag and drop RightInteractions to Hand in the second Hand Ref

Dominant Hand Game Object Filter Settings

Next, specify objects (fingertips) that can interact with UI in the Dominant Hand Game Object Filter attached to PalmMenu.

[Register left hand fingertip object]

• Open OVRInteractionComprehensive child element of OVRCameraRig object
• Further open child elements in this order: LeftInteractions -> Interactors -> Hand
• Confirm that there's HandPokeInteractor as a child element of Hand

• Click PalmMenu
• Drag and drop HandPokeInteractor to Element 0 directly under LeftHandedGameObjects in Dominant Hand Game Object Filter

[Register right hand fingertip object]

• Open OVRInteractionComprehensive child element of OVRCameraRig object
• Further open child elements in this order: RightInteractions -> Interactors -> Hand
• Confirm that there's HandPokeInteractor as a child element of Hand
• Click PalmMenu
• Drag and drop HandPokeInteractor to Element 0 directly under RightHandedGameObjects in Dominant Hand Game Object Filter

[Left Hand Settings]

In Left Hand of Dominant Hand Game Object Filter, you need to explicitly specify the Hand Ref associated with the left hand.

• Drag and drop PalmMenu to the area next to Left Hand
• When a dialog appears, select HandRef associated with LeftInteractions

4. Display Object Settings

When URP is selected, sample objects prepared may not display correctly due to material errors. Finally, adjust the sample objects.

• Create Material anywhere in the Assets folder
  This article sets the name as *CtrlObjMaterial
• Click ControlledObject child element of PalmMenu
• Drag and drop CtrlObjMaterial to the area marked Default-Material next to Element0 in MeshRenderer

5. Testing Operation

With the settings up to this point, you can operate the menu displayed on the left palm with fingers of the other hand. Also, menu display/hide can be toggled with tapping using the left thumb and index finger.

The PalmMenu prefab has pre-implemented object manipulation, but you can implement your own features by arranging this as a base. For details, refer to the Menu child element of PalmMenu.
For example, Unity's ScrollView is used for UI behavior, so you need to learn how to use this. Also, the PalmMenuExample script attached to the Menu object is helpful for more detailed behavior descriptions.

6. Bonus: Display Toggle by Palm Orientation

This time we toggled menu display with left thumb and index finger tapping, but you can also display the menu only when the palm faces toward you. I'll introduce this as an arrangement example of this sample.

Palm Orientation Detection Settings

• Pay attention to Menu child element of PalmMenu
• Disable OVR Button Active State
• Click Add Component at the bottom of Menu's Inspector
• Search for Transform Recognizer Active State and add
• Drag and drop LeftInteractions child element of OVRInteractionComprehensive to Hand in Transform Recognizer Active State
• Drag and drop Features child element of LeftInteractions to Transform Feature State Provider
• Change Transform Feature Configs to Palm Towards Face
• Open TransformConfig and set DefaultTransformFeatureStateThresholds.asset to Feature Thresholds *Search for DefaultTransformFeatureStateThresholds.asset in Project tab. (Select Search: All as search option)

Display/Hide Toggle Settings

• Find ActiveState Selector in Menu object's Inspector
• Drag and drop Menu object to Active State
• If the following dialog appears, select TransformRecognizerActiveState

• Find SelectorUnityEventWrapper in Menu object's Inspector
• Delete events set in When Unselected with - button
• Click + button for both When Selected and When Unselected
• Drag and drop Visuals child element of Menu object to None area in each event
• Click No Function in When Selected
• Click GameObject -> SetActive
• Check On
• Click No Function in When Unselected
• Click GameObject -> SetActive
• Check Off

Now the menu will only be displayed when the palm faces toward you.

7. Complete List of Meta XR SDK Articles

Meta XR SDK Series Index