Gesture recognition based on likelihood of interaction

Patent metadata

Field	Value
Publication number	US-11768544-B2
Application number	US-202217649659-A
Country	US
Kind code	B2
Filing date	Feb 1, 2022
Priority date	Feb 1, 2022
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

A method for evaluating gesture input comprises receiving input data for sequential data frames, including hand tracking data for hands of a user. A first neural network is trained to recognize features indicative of subsequent gesture interactions and configured to evaluate input data for a sequence of data frames and to output an indication of a likelihood of the user performing gesture interactions during a predetermined window of data frames. A second neural network is trained to recognize features indicative of whether the user is currently performing one or more gesture interactions and configured to adjust parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood. The second neural network evaluates the predetermined window for performed gesture interactions based on the adjusted parameters, and outputs a signal as to whether the user is performing one or more gesture interactions during the predetermined window.

First claim

The invention claimed is:

1. A method for evaluating gesture input, comprising:
receiving input data for sequential data frames, the input data including at least hand tracking data for one or more hands of a user;
at a first neural network trained to recognize features indicative of a plurality of subsequent gesture interactions:
evaluating input data for a sequence of two or more data frames; and
outputting an indication of a likelihood of the user performing one or more subsequent gesture interactions with a user interface during a predetermined window of one or more data frames;

at a second neural network trained to recognize features indicative of whether the user is currently performing one or more of the plurality of subsequent gesture interactions:
adjusting parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood of the user performing one or more of the plurality of subsequent gesture interactions;
evaluating at least the predetermined window for performed gesture interactions based on the adjusted parameters; and
outputting a signal as to whether the user is performing one or more gesture interactions with the user interface during the predetermined window.

2. The method of claim 1 , wherein the input data includes data generated by sensors of a head-mounted display worn by the user.

3. The method of claim 2 , wherein the input data further includes eye tracking data for the user.

4. The method of claim 3 , wherein the input data further includes body-pose data for the user.

5. The method of claim 2 , wherein the input data further includes display data regarding positioning of display objects of the user interface presented on the head-mounted display.

6. The method of claim 1 , wherein adjusting parameters includes decreasing a stringency of the parameters for gesture interaction recognition based on a likelihood of gesture interaction being above a likelihood threshold.

7. The method of claim 1 , wherein adjusting parameters includes increasing a stringency of the parameters for gesture interaction recognition based on a likelihood of gesture interaction being below a likelihood threshold.

8. The method of claim 1 , wherein adjusting parameters includes disabling gesture interaction recognition at the second neural network based on a likelihood of gesture interaction being below a likelihood threshold.

9. The method of claim 1 , wherein parameters for gesture interaction recognition during the predetermined window are adjusted based on the indicated likelihood of the user performing the gesture interaction being above a threshold certainty of the user being engaged in the gesture interaction during the sequence of two or more data frames.

10. The method of claim 1 , wherein the first neural network is trained to consider whether the user is manually interacting with a real-world object.

11. The method of claim 1 , wherein the first neural network is trained to consider whether a hand of the user is within a threshold distance of a nearest real-world surface.

12. The method of claim 1 , wherein the first neural network is trained to consider one or more indicators of hand tracking precision.

13. The method of claim 1 , wherein the first neural network is trained to consider whether the user is performing one or more repeatable pre-gesture behaviors.

14. The method of claim 1 , further comprising adjusting parameters for gesture recognition during the predetermined window based on one or more heuristic evaluations of input data for the predetermined window.

15. The method of claim 1 , wherein the predetermined window includes a current data frame.

16. The method of claim 1 , wherein the predetermined window includes one or more future data frames.

17. A system for a head-mounted display, comprising:
a display device configured to present a user interface to a user wearing the head-mounted display;
a sensor suite including at least one or more outward-facing image sensors;
one or more data pre-processing machines configured to receive data output from the sensor suite and to sequentially output data frames;
a first neural network trained to recognize features indicative of a plurality of subsequent gesture interactions, the first neural network configured to:
evaluate input data for a sequence of two or more data frames received from the one or more data pre-processing machines; and
output an indication of a likelihood of the user performing one or more subsequent gesture interactions with a user interface during a predetermined window of one or more data frames; and

a second neural network trained to recognize features indicative of whether the user is currently performing one or more of the plurality of subsequent gesture interactions, the second neural network configured to:
adjust parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood of the user performing one or more of the plurality of subsequent gesture interactions;
evaluate at least the predetermined window for performed gesture interactions based on the adjusted parameters; and
output a signal as to whether the user is performing one or more gesture interactions with the user interface during the predetermined window.

18. The system of claim 17 , further comprising one or more inward facing cameras, and wherein the input data further includes eye tracking data for the user.

19. The system of claim 17 , wherein the input data further includes user interface data regarding positioning of display objects on the user interface presented on the display device.

20. A method for evaluating gesture input, comprising:
receiving input data for sequential data frames, the input data including at least hand tracking data for one or more hands of a user;
based on an evaluation of the input data for a sequence of two or more data frames, outputting an indication of a likelihood of the user performing one or more subsequent gesture interactions with a user interface during a predetermined window of one or more data frames; and
at a neural network trained to recognize features indicative of whether the user is currently performing one or more of the plurality of subsequent gesture interactions:
adjusting parameters for gesture interaction recognition during the predetermined window based on the indicated likelihood of the user performing one or more of the plurality of subsequent gesture interactions, wherein adjusting parameters includes at least decreasing a stringency of the parameters for gesture interaction recognition based on a likelihood of gesture interaction being above a likelihood threshold and increasing a stringency of the parameters for gesture interaction recognition based on a likelihood of gesture interaction being below the likelihood threshold;
evaluating at least the predetermined window for performed gesture interactions based on the adjusted parameters; and
outputting a signal as to whether the user is performing one or more gesture interactions with the user interface during the predetermined window.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Schwarz Julia
• Tekin Bugra
• Stellmach Sophie
• Vazquez Erian
• Meekhof Casey Leon
• Gobel Fabian

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06N3/08

  Learning methods · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title