Artificial reality system having a sliding menu

Patent metadata

Field	Value
Publication number	US-10890983-B2
Application number	US-201916434919-A
Country	US
Kind code	B2
Filing date	Jun 7, 2019
Priority date	Jun 7, 2019
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. The artificial reality system can include a menu that can be activated and interacted with using one hand. In response to detecting a menu activation gesture performed using one hand, the artificial reality system can cause a menu to be rendered. A menu sliding gesture (e.g., horizontal motion) of the hand can be used to cause a slidably engageable user interface (UI) element to move along a horizontal dimension of the menu while horizontal positioning of the UI menu is held constant. Motion of the hand orthogonal to the menu sliding gesture (e.g., non-horizontal motion) can cause the menu to be repositioned. The implementation of the artificial reality system does require use of both hands or use of other input devices in order to interact with the artificial reality system.

First claim

What is claimed is:

1. An artificial reality system comprising:
an image capture device configured to capture image data;
a head mounted display (HMD) configured to output artificial reality content;
a gesture detector comprising processing circuitry configured to identify, from the image data, a menu activation gesture comprising a configuration of a hand in a substantially upturned orientation of the hand and a pinching configuration of a thumb and a finger of the hand and identify, from the image data and subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand;
a user interface (UI) engine comprising processing circuitry configured to, in response to the menu activation gesture, generate a menu interface and a slidably engageable UI element at a first position relative to the menu interface, and in response to the menu sliding gesture, translate the slidably engageable UI element to a second position relative to the menu interface; and
a rendering engine comprising processing circuitry configured to render the artificial reality content, the menu interface, and the translation of the slidably engageable UI element from the first position relative to the user interface to the second position relative to the user interface for display at the HMD.

2. The artificial reality system of claim 1 ,
wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, and
wherein the UI engine is configured to highlight one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.

3. The artificial reality system of claim 2 , wherein the one or more menu items correspond to respective applications executing on the artificial reality system.

4. The artificial reality system of claim 1 ,
wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface,
wherein to translate the slidably engageable UI element to the second position relative to the menu interface, the UI engine is configured to slide the slidably engageable UI element along the dimension of the menu interface to the second position relative to the menu interface.

5. The artificial reality system of claim 1 ,
wherein the menu sliding gesture comprises motion of the hand in a substantially first direction,
wherein the gesture detector is configured to identify, from the image data, motion of the hand in a substantially second direction subsequent to the menu activation gesture, the substantially second direction being substantially orthogonal to the substantially first direction,
wherein the UI engine is further configured to, in response to the motion of the hand in the substantially second direction, translate the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.

6. The artificial reality system of claim 1 , further comprising:
an application engine comprising processing circuitry for execution of one or more artificial reality applications,
wherein the gesture detector is configured to identify, from the image data, a selection gesture subsequent to the menu sliding gesture, and
wherein the application engine is configured to perform an action in response to the selection gesture.

7. The artificial reality system of claim 6 , wherein the selection gesture comprises one of (1) movement of a different finger of the hand, (2) translation of the hand in a direction that is substantially normal to the menu interface, or (3) reconfiguring the thumb and the finger of the hand to no longer be in the pinching configuration.

8. The artificial reality system of claim 1 ,
wherein the gesture detector is further configured to identify, from the image data, a menu prompt gesture prior to the menu activation gesture, and
wherein the UI engine is further configured to generate a menu prompt element in response to the menu prompt gesture.

9. The artificial reality system of claim 8 , wherein the menu prompt gesture comprises the hand configured in a substantially upturned position with a space between a thumb and a finger, and
wherein the UI engine generates the menu prompt element in the space between a thumb and a finger of a virtual hand.

10. The artificial reality system of claim 9 , wherein the menu prompt element comprises a line between the thumb and the finger of the virtual hand.

11. The artificial reality system of claim 1 , wherein the image capture device is integrated with the HMD.

12. A method comprising:
obtaining, by an artificial reality system including a head mounted display (HMD), image data via an image capture device;
identifying, by the artificial reality system from the image data, a menu activation gesture, the menu activation gesture comprising a configuration of a hand in a substantially upturned orientation of the hand and a pinching configuration of a thumb and a finger of the hand;
generating, by the artificial reality system in response to the menu activation gesture, a menu interface and a slidably engageable UI element at a first position relative to the menu interface;
identifying, from the image data and subsequent to the menu activation gesture, a menu sliding gesture comprising the configuration of the hand in combination with a motion of the hand;
in response to the menu sliding gesture, translate the slidably engageable UI element to a second position relative to the menu interface, and
rendering, by the artificial reality system, artificial reality content, the menu interface, and the translation of the slidably engageable UI element from the first position relative to the menu interface to the second position relative to the user interface for display at the HMD.

13. The method of claim 12 , wherein the menu interface comprises one or more menu items arrayed along a dimension of the menu interface, the method further comprising:
highlighting, by the artificial reality system, one of the menu items according to a position of the slidably engageable UI element relative to the menu interface.

14. The method of claim 12 , wherein translating the slidably engageable UI element to the second position relative to the menu interface comprises sliding the slidably engageable UI element along a dimension of the menu interface to the second position relative to the menu interface.

15. The method of claim 12 ,
wherein the menu sliding gesture comprises motion of the hand in a substantially first direction, the method further comprising:
identifying, by the artificial reality system from the image data, motion of the hand in a substantially second direction subsequent to the menu activation gesture, the substantially second direction being substantially orthogonal to the substantially first direction; and
translating, by the artificial reality system in response to the motion of the hand in the substantially second direction, the slidably engageable UI element and the menu interface while retaining a position of the slidably engageable UI element relative to the menu interface.

16. The method of claim 12 , further comprising:
identifying, by the artificial reality system from the image data, a menu prompt gesture; and
generating, by the artificial reality system, a menu prompt element in response to the menu prompt gesture.

17. A non-transitory, computer-readable medium comprising instructions that, when executed, cause one or more processors of

Assignees

• Facebook Tech Llc

Inventors

• Ravasz Jonathan
• Stevens Jasper
• Varga Adam Tibor
• Pinchon Etienne
• Tickner Simon Charles
• Spurlock Jennifer Lynn
• Sorge-Toomey Kyle Eric
• Ellis Robert
• Fox Barrett

Classifications

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F3/0482 Primary

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/0304 Primary

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title