Input recognition based on distinguishing direct and indirect user interactions

Patent metadata

Field	Value
Publication number	US-12443286-B2
Application number	US-202318375280-A
Country	US
Kind code	B2
Filing date	Sep 29, 2023
Priority date	Jun 2, 2023
Publication date	Oct 14, 2025
Grant date	Oct 14, 2025

Abstract

Various implementations disclosed herein include devices, systems, and methods that interpret user activity as user interactions with user interface (UI) elements positioned within a three-dimensional (3D) space such as an extended reality (XR) environment. Some implementations enable user interactions with virtual elements displayed in 3D environments that utilize alternative input modalities, e.g., XR environments that interpret user activity as either direct interactions or indirect interactions with virtual elements.

First claim

What is claimed is:

1. A method comprising:
at an electronic device having a processor:
obtaining a position of a virtual object in an extended reality (XR) environment corresponding to a three-dimensional (3D) space;
obtaining a user hand position in the 3D space based on sensor data, wherein the hand position is associated with a hand gesture;
determining an interaction mode based on the user hand position and the position of the virtual object in the 3D space, wherein a direct interaction mode is selected when a relationship between the user hand position and the position of the virtual object in the 3D space satisfies a criterion and an indirect interaction mode is selected when the relationship satisfies a second criterion different than the first criterion; and
interpreting the hand gesture using an interaction recognition process associated with the determined interaction mode, wherein a direct interaction recognition process is used when the direct interaction mode is selected and an indirect interaction recognition process is used when the indirect interaction mode is selected.

2. The method of claim 1 ,
wherein determining the interaction mode comprises selecting a direct interaction mode in which the virtual object is identified based on the hand position intersecting a 3D area associated with the virtual object in the 3D space, and
wherein the interaction recognition process is a direct interaction recognition process applied to the hand gesture.

3. The method of claim 1 , wherein determining the interaction mode comprises selecting a direct interaction mode based on the hand position intersecting a 3D area associated with the virtual object in the 3D space.

4. The method of claim 1 , wherein determining the interaction mode comprises selecting a direct interaction mode based on:
determining an attention zone based on a gaze direction or head pose of the user;
determining an intersection of the hand position intersecting a 3D area associated with the virtual object in the 3D space; and
determining that the intersection is within the attention zone.

5. The method of claim 1 , wherein the interaction mode is a direct interaction mode and the method further comprises discontinuing interaction detection for a predetermined period of time following a recognition of the hand gesture.

6. The method of claim 1 further comprising initiating a selection of the virtual object based on interaction mode being a direct interaction mode and the hand gesture being a direct touch with the virtual object.

7. The method of claim 1 further comprising initiating a hover of the virtual object based on interaction mode being a direct interaction mode and the hand position satisfying a proximity criterion with respect to the position of the virtual object and within an attention zone.

8. The method of claim 7 further comprising maintaining the hover of the virtual object when the position of the virtual object is no longer in the attention zone.

9. The method of claim 1 ,
wherein determining the interaction mode comprises selecting an indirect interaction mode in which the virtual object is identified based on a gaze direction intersecting an area associated with the virtual object, and
wherein the interaction recognition process is an indirect interaction recognition process applied to the hand gesture.

10. The method of claim 1 , wherein determining the interaction mode comprises selecting an indirect interaction mode based on the hand position being outside of a 3D area associated with the virtual object in the 3D space.

11. The method of claim 1 , wherein determining the interaction mode comprises selecting an indirect interaction mode based on:
determining an attention zone based on a gaze direction or head pose of the user;
determining an intersection of the hand position intersecting a 3D area associated with the virtual object in the 3D space; and
determining that the intersection is outside of the attention zone.

12. The method of claim 1 further comprising initiating a selection of the virtual object based on interaction mode being an indirect interaction mode and the hand gesture being a pinch occurring while a gaze is direct towards the virtual object.

13. The method of claim 1 , wherein the interaction mode is an indirect interaction mode and interpreting the hand gesture comprises:
recognizing that the hand gesture is a pinch; and
identifying the virtual object based on a gaze direction occurring during the pinch.

14. The method of claim 1 , wherein the interaction mode is an indirect interaction mode and interpreting the hand gesture comprises:
recognizing that the hand gesture corresponds to hand engagement;
identifying the virtual object based on a gaze direction occurring during the hand engagement; and
initiating a hover of the virtual object based on the hand engagement.

15. The method of claim 1 further comprising:
identifying a hand gesture;
tracking a position of the hand gesture in the 3D space; and
identifying one or more interaction points based on the tracking.

16. The method of claim 15 , wherein identifying the one or more interaction points is based on a gaze direction.

17. The method of claim 1 , wherein the electronic device is a head-mounted device and the XR environment is a virtual reality environment or an augmented reality environment.

18. The method of claim 1 , further comprising providing data corresponding to user activity to an application for the application to interpret the user activity based on the data, wherein the data is specific to the interaction mode.

19. The method of claim 18 , wherein for a direct interaction mode, the data comprises an index finger tip as a manipulator pose and a trajectory corrected interaction pose.

20. The method of claim 18 , wherein for an indirect interaction mode, the data comprises a pinch centroid as a manipulator pose and an interaction pose or user-centric data.

21. A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
obtaining a position of a virtual object in an extended reality (XR) environment corresponding to a three-dimensional (3D) space;
obtaining a user hand position in the 3D space based on sensor data, wherein the hand position is associated with a hand gesture;
determining an interaction mode based on the user hand position and the position of the virtual object in the 3D space, wherein a direct interaction mode is selected when a relationship between the user hand position and the position of the virtual object in the 3D space satisfies a criterion and an indirect interaction mode is selected when the relationship satisfies a second criterion different than the first criterion; and
interpreting the hand gesture using an interaction recognition process associated with the determined interaction mode, wherein a direct interaction recognition process is used when the direct interaction mode is selected and an indirect interaction recognition process is used when the indirect interaction mode is selected.

22. The system of claim 21 ,
wherein determining the interaction mode comprises selecting a direct interaction mode in which the virtual element is ide

Assignees

• Apple Inc

Inventors

• Meyer David J
• Shutzberg Julian K
• Teitelbaum David M
• Brewer Daniel J
• Dandu Bharat C
• Mckenzie Christopher D

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06V40/18

  Eye characteristics, e.g. of the iris · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title