Artificial reality systems with personal assistant element for gating user interface elements

Patent metadata

Field	Value
Publication number	US-10921879-B2
Application number	US-201916435094-A
Country	US
Kind code	B2
Filing date	Jun 7, 2019
Priority date	Jun 7, 2019
Publication date	Feb 16, 2021
Grant date	Feb 16, 2021

Abstract

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. The artificial reality system includes an image capture device, a head-mounted display (HMD), a user interface (UI) engine, and a rendering engine. The image capture device captures image data representative of a physical environment. The HMD outputs artificial reality content, the artificial reality content including an assistant element. The gesture detector identifies, from the image data, a gesture that includes a gripping motion of two or more digits of a hand to form a gripping configuration at a location that corresponds to the assistant element, and subsequent to the gripping motion, a throwing motion of the hand with respect to the assistant element. The UI engine generates a UI element in response to identifying the gesture.

First claim

What is claimed is:

1. An artificial reality system comprising:
an image capture device configured to capture image data representative of a physical environment;
a head-mounted display (HMD) configured to output artificial reality content, the artificial reality content including an assistant element;
a gesture detector configured to:
identify, from the image data, a first gesture comprising:
a gripping motion of two or more digits of a hand to form a gripping configuration at a location that corresponds to the assistant element, and
subsequent to the gripping motion, a throwing motion of the hand with respect to the assistant element; and

identify, from the image data, a press-and-hold gesture with respect to the assistant element;

a user interface (UI) engine configured to generate a UI element in response to identifying the first gesture; and
a rendering engine configured to:
render the UI element as an overlay to at least some of the artificial reality content; and
remove the overlay of the UI element from the artificial reality content in response to identifying the press-and-hold gesture, the press-and-hold gesture occurring after the rendering of the UI element as the overlay.

2. The artificial reality system of claim 1 , wherein the gripping motion comprises a thumb of the hand at a location that corresponds to a first portion of the assistant element and at least one finger of the hand other than the thumb at a location that corresponds to a second portion of the assistant element.

3. The artificial reality system of claim 2 , wherein the first portion of the assistant element is at least approximately diametrically opposed to the second portion of the assistant element.

4. The artificial reality system of claim 1 , wherein the throwing motion comprises a combination of a release of the gripping configuration and a flexion of the hand.

5. The artificial reality system of claim 1 , wherein the throwing motion comprises a combination of a release of the gripping configuration and an outward flicking motion of at least one of the digits of the hand.

6. The artificial reality system of claim 1 , wherein the UI element comprises at least one menu of user-selectable options.

7. The artificial reality system of claim 1 , wherein, to identify the press-and-hold gesture, the gesture detector is configured to identify a configuration of at least one of the digits of the hand positioned at a location that corresponds to the assistant element in the artificial reality content and that is substantially stationary for at least a threshold period of time.

8. The artificial reality system of claim 1 , wherein the gesture detector is configured to identify the first gesture as corresponding to a first entry in a gesture library, and wherein the gesture detector is configured to identify the press-and-hold gesture as corresponding to a second entry in the gesture library.

9. The artificial reality system of claim 1 , wherein the image capture device is integrated within the HMD.

10. A method comprising:
capturing, by a head-mounted display (HMD), image data representative of a physical environment;
outputting, by the HMD, artificial reality content, the artificial reality content including an assistant element;
identifying, by a gesture detector, from the image data, a first gesture comprising:
a gripping motion using two or more digits of a hand at a location that corresponds to the assistant element, and
subsequent to the gripping motion, a throwing motion of the hand with respect to the assistant element;

identifying, by the gesture detector, from the image data, a press-and-hold gesture with respect to the assistant element;
generating, by a user interface (UI) engine, a UI element in response to identifying the first gesture;
rendering, by a rendering engine, the UI element as an overlay to at least some of the artificial reality content; and
removing, by the rendering engine, the overlay of the UI element from the artificial reality content in response to identifying the press-and-hold gesture, the press-and-hold gesture occurring after the rendering of the UI element as the overlay.

11. The method of claim 10 , wherein the gripping motion comprises a thumb of the hand contacting a first portion of the assistant element and at least one finger of the hand other than the thumb contacting a second portion of the assistant element.

12. The method of claim 11 , wherein the first portion of the assistant element is at least approximately diametrically opposed to the second portion of the assistant element.

13. The method of claim 10 , wherein the throwing motion comprises a combination of a release of the gripping motion and a flexion of the hand.

14. The method of claim 10 , wherein the throwing motion comprises a combination of a release of the gripping motion and an outward flicking motion of at least one of the digits of the hand.

15. The method of claim 10 , wherein the UI element comprises at least one menu of user-selectable options.

16. The method of claim 10 , wherein identifying the press-and-hold gesture comprises determining, by the gesture detector, that at least one of the digits of the hand is positioned at a location that corresponds to the assistant element in the artificial reality content and substantially stationary for at least a threshold period of time.

17. The method of claim 10 , wherein identifying the first gesture comprises identifying, by the gesture detector, the first gesture as corresponding to a first entry in a gesture library, and wherein identifying the press-and-hold gesture comprises identifying, by the gesture detector, the press-and-hold gesture as corresponding to a second entry in the gesture library.

18. A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause processing circuitry of an artificial reality system to:
receive, from a head-mounted display (HMD), image data representative of a physical environment;
output, via the HMD, artificial reality content including an assistant element;
identify, from the image data, a first gesture comprising:
a gripping motion using two or more digits of a hand at a location that corresponds to the assistant element, and
subsequent to the gripping motion, a throwing motion of the hand with respect to the assistant element;

identify, from the image data, a press-and-hold gesture with respect to the assistant element;
generate a UI element in response to identifying the first gesture;
render the UI element as an overlay to at least some of the artificial reality content; and
remove the overlay of the UI element from the artificial reality content in response to identifying the press-and-hold gesture, the press-and-hold gesture occurring after the rendering of the UI element as the overlay.

Assignees

• Facebook Tech Llc

Inventors

• Ravasz Jonathan
• Stevens Jasper
• Varga Adam Tibor
• Pinchon Etienne
• Tickner Simon Charles
• Spurlock Jennifer Lynn
• Sorge-Toomey Kyle Eric
• Ellis Robert
• Fox Barrett

Classifications

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title