Corner-identifiying gesture-driven user interface element gating for artificial reality systems

Patent metadata

Field	Value
Publication number	US-11043192-B2
Application number	US-201916435079-A
Country	US
Kind code	B2
Filing date	Jun 7, 2019
Priority date	Jun 7, 2019
Publication date	Jun 22, 2021
Grant date	Jun 22, 2021

Abstract

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. The artificial reality system includes an image capture device, a head-mounted display (HMD), a gesture detector, a user interface (UI) engine, and a rendering engine. The image capture device captures image data representative of a physical environment. The HMD outputs artificial reality content. The gesture detector identifies, from the image data, a gesture including a configuration of a hand that is substantially stationary for at least a threshold period of time and positioned such that an index finger and a thumb of the hand form approximately a right angle. The UI engine generates a UI element in response to the identified gesture. The rendering engine renders the UI element as an overlay to the artificial reality content.

First claim

What is claimed is:

1. An artificial reality system comprising:
an image capture device configured to capture image data representative of a physical environment;
a head-mounted display (HMD) configured to output artificial reality content;
a gesture detector configured to:
identify, from the image data, a first gesture comprising a configuration of a hand that is substantially stationary for at least a threshold period of time and positioned such that an index finger and a thumb of the hand form approximately a right angle at a first hand orientation, and
subsequently identify, from the image data, a second gesture comprising the configuration of the hand such that the index finger and the thumb of the hand form approximately the right angle at a second hand orientation different than the first hand orientation; and

a user interface (UI) engine configured to:
generate a UI menu conforming to a first UI orientation indicated by the first hand orientation in response to the identified first gesture, the UI menu, when in the first UI orientation, comprising a first data granularity-level of user-selectable options, and
generate the UI menu conforming to a second UI orientation indicated by the second hand orientation in response to the identified second gesture, the UI menu, when in the second UI orientation, comprising a second data granularity-level of user-selectable options, wherein the second data granularity-level comprises a finer-grained set of user-selectable options than the first data granularity-level; and

a rendering engine configured to render the UI menu as an overlay to the artificial reality content.

2. The artificial reality system of claim 1 , wherein the second hand orientation of the second gesture represents a 90-degree hand rotation and a 180-degree hand reflection with respect to the first hand orientation of the first gesture.

3. The artificial reality system of claim 1 ,
wherein the first UI orientation is a portrait orientation, and
wherein the second UI orientation is a landscape orientation.

4. The artificial reality system of claim 1 , wherein the UI menu, when in the second UI orientation, comprises a full set of user-selectable options, and wherein the UI menu, when in the first UI orientation, comprises a subset of the full set of user-selectable options.

5. The artificial reality system of claim 1 , wherein each of the identified first gesture or the identified second gesture indicates a location of a corner of the UI menu and the first UI orientation or the second UI orientation, respectively, of the UI menu within the artificial reality content.

6. The artificial reality system of claim 5 , wherein a corner of the right angle formed by the index finger and thumb of the hand identified from the image data indicates the location of the corner of the UI menu within the artificial reality content.

7. The artificial reality system of claim 5 , wherein a position of the index finger of the hand identified from the image data indicates one of the first UI orientation or the second UI orientation of the UI menu within the artificial reality content, wherein a long side of the UI menu substantially aligns with the index finger of the hand.

8. The artificial reality system of claim 1 , wherein to identify the first gesture, the gesture detector is configured to identify the first gesture as being represented by a first entry in a gesture library, and wherein the identify the second gesture, the gesture detector is configured to identify the second gesture as being represented by a second entry in the gesture library.

9. The artificial reality system of claim 1 , wherein the image capture device is integrated within the HMD.

10. A method comprising:
outputting, by a head-mounted display (HMD), artificial reality content;
capturing, by the HMD, image data representative of a physical environment;
identifying, by a gesture detector, from the image data, a first gesture comprising a configuration of a hand that is substantially stationary for at least a threshold period of time and positioned such that an index finger and a thumb of the hand form approximately a right angle at a first hand orientation;
generating, by a user interface (UI) engine, a UI menu conforming to a first UI orientation indicated by the first hand orientation in response to the identified first gesture, the UI menu, when in the first UI orientation, comprising a first data granularity-level of user-selectable options;
rendering, by a rendering engine, the UI menu in the first UI orientation as an overlay to the artificial reality content;
subsequently identifying, by the gesture detector, from the image data, a second gesture comprising the configuration of the hand such that the index finger and the thumb of the hand form approximately the right angle at a second hand orientation different than the first hand orientation;
generating, by the UI engine, the UI menu conforming to a second UI orientation indicated by the second hand orientation in response to the identified second gesture, the UI menu, when in the second UI orientation, comprising a second data granularity-level of user-selectable options, wherein the second data granularity-level comprises a finer-grained set of user-selectable options than the first data granularity-level; and
rendering, by a rendering engine, the UI menu in the second orientation as the overlay to the artificial reality content.

11. The method of claim 10 , wherein the second hand orientation of the second gesture represents a 90-degree hand rotation and a 180-degree hand reflection with respect to the first hand orientation of the first gesture.

12. The method of claim 10 ,
wherein the first UI orientation is a portrait orientation, and
wherein the second UI orientation is a landscape orientation.

13. The method of claim 10 , wherein the UI menu, when in the second UI orientation, comprises a full set of user-selectable options, and wherein the UI menu, when in the first UI orientation, comprises a subset of the user-selectable options.

14. The method of claim 10 , wherein each of the identified first gesture or the identified second gesture indicates a location of a corner of the UI menu and the first UI orientation or the second UI orientation, respectively, of the UI menu within the artificial reality content.

15. The method of claim 14 , wherein a corner of the right angle formed by the index finger and thumb of the hand identified from the image data indicates the location of the corner of the UI menu within the artificial reality content.

16. The method of claim 14 , wherein a position of the index finger of the hand identified from the image data indicates one of the first UI orientation or the second UI orientation of the UI menu within the artificial reality content, wherein a long side of the UI menu substantially aligns with the index finger of the hand.

17. The method of claim 10 , wherein identifying the first gesture comprises identifying, by the gesture detector, the first gesture as being represented by a first entry in a gesture library, and wherein identifying the second gesture comprises identifying, by the gesture detector, the second gesture as being represented by a second entry in the gesture library.

18. A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause processing circuitry of an artificial reality system to:
output artificial reality content via by a head-mounted display (HMD);
receive image data representative of a physical environm

Assignees

• Facebook Tech Llc

Inventors

• Ravasz Jonathan
• Stevens Jasper
• Varga Adam Tibor
• Pinchon Etienne
• Tickner Simon Charles
• Spurlock Jennifer Lynn
• Sorge-Toomey Kyle Eric
• Ellis Robert
• Fox Barrett

Classifications

• G09G2354/00

  Aspects of interface with display user · CPC title

• G09G5/377 Primary

  for mixing or overlaying two or more graphic patterns (G09G5/02, G09G5/397 take precedence) · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title