Artificial reality system having a digit-mapped self-haptic input method

Patent metadata

Field	Value
Publication number	US-10955929-B2
Application number	US-201916435139-A
Country	US
Kind code	B2
Filing date	Jun 7, 2019
Priority date	Jun 7, 2019
Publication date	Mar 23, 2021
Grant date	Mar 23, 2021

Abstract

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. The artificial reality system captures image data representative of a physical environment and outputs the artificial reality content. The artificial reality system identifies, from the image data, a gesture comprising a motion of a first digit of a hand and a second digit of the hand to form a pinching configuration a particular number of times within a threshold amount of time. The artificial reality system assigns one or more input characters to one or more of a plurality of digits of the hand and processes a selection of a first input character of the one or more input characters assigned to the second digit of the hand in response to the identified gesture.

First claim

What is claimed is:

1. An artificial reality system comprising:
an image capture device configured to capture image data representative of a physical environment;
a head-mounted display (HMD) configured to output artificial reality content;
a gesture detector comprising processing circuitry configured to identify, from the image data, a gesture comprising a motion of a first digit of a hand and a second digit of the hand to form a pinching configuration a particular number of times within a threshold amount of time; and
a user interface engine comprising processing circuitry configured to:
assign one or more input characters to one or more of a plurality of digits of the hand, wherein the one or more input characters assigned to the second digit of the hand comprise a distinct set of input characters,
map each of the one or more input characters in the distinct set of input characters assigned to the second digit of the hand to a respective selection number, wherein the respective selection number is less than or equal to a cardinality of the distinct set, and
process a selection of a first input character of the one or more input characters in the distinct set of input characters assigned to the second digit of the hand based on the selection number mapped to the first input character in response to the particular number of times the pinching configuration is formed in the identified gesture.

2. The artificial reality system of claim 1 , further comprising a rendering engine comprising processing circuitry configured to render the one or more input characters assigned to the one or more of the plurality of digits of the hand as an overlay to an image of the hand.

3. The artificial reality system of claim 1 , wherein, to assign the one or more input characters, the user interface engine is configured to assign a distinct set of input characters to each digit of the one or more of the plurality of digits of the hand.

4. The artificial reality system of claim 3 , wherein the hand comprises a first hand, and wherein the user interface engine is further configured to assign a distinct set of input characters to each digit of one or more of a plurality of digits of a second hand.

5. The artificial reality system of claim 1 , wherein, to process the selection of the first input character, the user interface engine is configured to:
determine the selection of the first input character based on the selection number mapped to the first input character being equal to the particular number of times the first digit of the hand and the second digit of the hand form the pinching configuration within the threshold amount of time for the identified gesture.

6. The artificial reality system of claim 1 , wherein, to process the selection of the first input character, the user interface engine is configured to:
calculate a quotient with a remainder by dividing the particular number of times the first digit of the hand and the second digit of the hand form the pinching configuration within the threshold amount of time for the identified gesture by the cardinality of the distinct set; and
determine the selection of the first input character based on the selection number mapped to the first input character being equal to the remainder.

7. The artificial reality system of claim 1 , further comprising a rendering engine comprising processing circuitry configured to render, during the gesture, a current input character of the one or more input characters in the distinct set of input characters assigned to the second digit of the hand that would be selected based on a current number of times the first digit of the hand and the second digit of the hand form the pinching configuration within the threshold period of time.

8. The artificial reality system of claim 1 , wherein the gesture comprises a first gesture,
wherein the gesture detector is further configured to identify, from the image data, a second gesture; and
wherein the user interface engine is configured to assign one or more special input characters to the second gesture, and process a selection of a first special input character of the one or more special input characters assigned to the second gesture in response to the identified second gesture.

9. The artificial reality system of claim 1 , wherein the one or more input characters comprise one or more of letters, numbers, symbols, or a null character.

10. The artificial reality system of claim 1 , wherein the image capture device is integrated within the HMD.

11. A method comprising:
capturing, by an image capture device of an artificial reality system, image data representative of a physical environment;
outputting, by head mounted display (HMD) of the artificial reality system, artificial reality content;
identifying, from the image data, a gesture comprising a motion of a first digit of a hand and a second digit of the hand to form a pinching configuration a particular number of times within a threshold amount of time;
assigning one or more input characters to one or more of a plurality of digits of the hand, wherein the one or more input characters assigned to the second digit of the hand comprise a distinct set of input characters;
mapping each of the one or more input characters in the distinct set of input characters assigned to the second digit of the hand to a respective selection number, wherein the respective selection number is less than or equal to a cardinality of the distinct set; and
processing a selection of a first input character of the one or more input characters in the distinct set of input characters assigned to the second digit of the hand based on the selection number mapped to the first input character in response to the particular number of times the pinching configuration is formed in the identified gesture.

12. The method of claim 11 , further comprising rendering the one or more input characters assigned to the one or more of the plurality of digits of the hand as an overlay to an image of the hand.

13. The method of claim 11 , wherein the hand comprises a first hand, and wherein assigning the one or more input characters comprises:
assigning a distinct set of input characters to each digit of the one or more of the plurality of digits of the first hand; and
assigning a distinct set of input characters to each digit of one or more of a plurality of digits of a second hand.

14. The method of claim 11 , wherein processing the selection of the first input character comprises:
determining the selection of the first input character based on the selection number mapped to the first input character being equal to the particular number of times the first digit of the hand and the second digit of the hand form the pinching configuration within the threshold amount of time for the identified gesture.

15. The method of claim 11 , wherein processing the selection of the first input character comprises:
calculating a quotient with a remainder by dividing the particular number of times the first digit of the hand and the second digit of the hand form the pinching configuration within the threshold amount of time for the identified gesture by the cardinality of the distinct set; and
determining the selection of the first input character based on the selection number mapped to the first input character being equal to the remainder.

16. The method of claim 11 , further comprising rendering, during the gesture, a current input character of the one or more input characters in the distinct set of input characters assigned to the second digit of the hand that would be selected

Assignees

• Facebook Tech Llc

Inventors

• Ravasz Jonathan
• Stevens Jasper
• Varga Adam Tibor
• Pinchon Etienne
• Tickner Simon Charles
• Spurlock Jennifer Lynn
• Sorge-Toomey Kyle Eric
• Ellis Robert
• Fox Barrett

Classifications

• G06F3/04886

  by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus · CPC title

• G06F3/0484

  for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• G06F3/0485

  Scrolling or panning · CPC title