Detecting input in artificial reality systems based on a pinch and pull gesture

Patent metadata

Field	Value
Publication number	US-12099693-B2
Application number	US-202217658982-A
Country	US
Kind code	B2
Filing date	Apr 12, 2022
Priority date	Jun 7, 2019
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. In one example, an artificial reality system comprises an image capture device configured to capture image data representative of a physical environment; a head-mounted display (HMD) configured to output artificial reality content; a gesture detector configured to identify, from the image data, a gesture comprising a motion of two fingers from a hand to form a pinching configuration and a subsequent pulling motion while in the pinching configuration; a user interface (UI) engine configured to generate a UI input element in response to identifying the gesture; and a rendering engine configured to render the UI input element as an overlay to at least some of the artificial reality content.

First claim

What is claimed is:

1. An artificial reality system comprising:
an image capture system configured to capture image data representative of a physical environment;
a head-mounted display (HMD);
a gesture detection engine configured to identify, from the image data, a gesture comprising a motion of two fingers from a hand to form a pinching configuration in a vicinity of a UI pinch element, the vicinity being at or within a threshold distance of the UI pinch element, and a subsequent pulling motion towards the HMD while in the pinching configuration; and a user interface (UI) engine configured to:
generate a UI input element, including a string element, in response to identifying the gesture, wherein the string element extends from the UI pinch element to a default position on the UI input element, modify the UI input element, including the string element, in response to further movements of the hand while in the pinching configuration, and
render the modified UI input element, including the modified string element, as a three-dimensional overlay to artificial reality content presented by the HMD,

wherein the modified string element extends from a location at or within a threshold distance of the UI pinch element to a location at or within a threshold distance of the two fingers from the hand.

2. The artificial reality system of claim 1 , wherein the UI input element has a size, and wherein to modify the UI input element in response to the further movements of the hand while in the pinching configuration, the UI engine is further configured to:
change the size of the UI input element.

3. The artificial reality system of claim 1 , wherein the UI input element has an appearance, and
wherein to modify the UI input element in response to the further movements of the hand while in the pinching configuration, the UI engine is further configured to: change the appearance of the UI input element.

4. The artificial reality system of claim 1 , wherein the UI input element has a color, wherein the further movements of the hand while in the pinching configuration occur within three-dimensional space, and
wherein to modify the UI input element in response to the further movements of the hand, the UI engine is further configured to:
change the color of the UI input element.

5. The artificial reality system of claim 1 , wherein the gesture detection engine is further configured to: determine, from the image data, that the pinching configuration has ceased.

6. The artificial reality system of claim 5 , wherein the UI engine is further configured to:
responsive to determining that the pinching configuration has ceased, remove the UI input element from the artificial reality content presented by the HMD.

7. The artificial reality system of claim 5 , wherein the UI engine is further configured to responsive to determining that the pinching configuration has ceased, identify an input value; and generate, based on the input value, updated artificial reality content to be displayed by the HMD.

8. The artificial reality system of claim 1 , wherein the two fingers from the hand are a thumb and index finger.

9. The artificial reality system of claim 1 , wherein the subsequent pulling motion includes a pulling motion of the hand.

10. The artificial reality system of claim 1 , wherein the subsequent pulling motion is a pulling motion of the two fingers.

11. A method comprising:
capturing, by an image capture system included within an artificial reality system, image data representative of a physical environment;
outputting, by a head-mounted display (HMD) included within the artificial reality system, artificial reality content;
identifying, by the artificial reality system and from the image data, a gesture comprising a motion of two fingers from a hand to form a pinching configuration in a vicinity of a UI pinch element, the vicinity being at or within a threshold distance of the UI pinch element, and a subsequent pulling motion towards the HMD while in the pinching configuration;
generating, by the artificial reality system, a UI input element, including a string
element, in response to identifying the gesture, wherein the string element extends from the UI pinch element to a default position on the UI input element;
modifying, by the artificial reality system, the UI input element, including the string element, in response to further movements of the hand while in the pinching configuration; and
rendering, by the artificial reality system, the modified UI input element, including the modified string element, as a three-dimensional overlay to the artificial reality content presented by the HMD,
wherein the modified string element extended from the UI pinch element to a location at or within a threshold distance of the two fingers from the hand based on the further movements of the hand while in the pinching configuration.

12. The method of claim 11 , wherein the UI input element has a size, and wherein modifying the UI input element in response to the further movements of the hand while in the pinching configuration includes: changing the size of the UI input element.

13. The method of claim 11 , wherein modifying the UI input element in response to the further movements of the hand while in the pinching configuration includes: changing the appearance of the UI input element.

14. The method of The method of claim 11 , wherein the UI input element has a color,
wherein the further movements of the hand while in the pinching configuration occur within three-dimensional space, and
wherein modifying the UI input element in response to the further movements of the hand includes:
changing the color of the UI input element.

15. A non-transitory computer-readable medium comprising instructions for causing one or more processors of an artificial reality system to perform operations comprising:
capturing image data representative of a physical environment; outputting, by a head-mounted display (HMD) included within the artificial reality system, artificial reality content;
identifying, from the image data, a gesture comprising a motion of two fingers from a hand to form a pinching configuration in a vicinity of a UI pinch element, the vicinity being at or within a threshold distance of the UI pinch element, and a subsequent pulling motion towards the HMD while in the pinching configuration;
generating a UI input element, including a string element, in response to identifying the gesture, wherein the string element extends from the UI pinch element to a default position on the UI input element;
modifying the UI input element, including the modified string element, in response to further movements of the hand while in the pinching configuration; and
rendering the UI input element as a three-dimensional overlay to the artificial reality content presented by the HMD,
wherein modified string element extends from the UI pinch element to a location at or within a threshold distance of the two fingers from the hand based on the further movements of the hand while in the pinching configuration.

Assignees

• Meta Platforms Tech Llc

Inventors

• Ravasz Jonathan
• Stevens Jasper
• Varga Adam Tibor
• Pinchon Etienne
• Tickner Simon Charles
• Spurlock Jennifer Lynn
• Sorge-Toomey Kyle Eric
• Ellis Robert
• Fox Barrett

Classifications

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G02B27/017

  Head mounted · CPC title

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title