Augmented two-stage hand gesture input

Patent metadata

Field	Value
Publication number	US-11294472-B2
Application number	US-201916365114-A
Country	US
Kind code	B2
Filing date	Mar 26, 2019
Priority date	Jan 11, 2019
Publication date	Apr 5, 2022
Grant date	Apr 5, 2022

Abstract

A method for augmenting a two-stage hand gesture input comprises receiving hand tracking data for a hand of a user. A gesture recognition machine recognizes that the user has performed a first-stage gesture based on one or more parameters derived from the received hand tracking data satisfying first-stage gesture criteria. An affordance cueing a second-stage gesture is provided to the user responsive to recognizing the first-stage gesture. The gesture recognition machine recognizes that the user has performed the second-stage gesture based on one or more parameters derived from the received hand tracking data satisfying second-stage gesture criteria. A graphical user interface element is displayed responsive to recognizing the second-stage gesture.

First claim

The invention claimed is:

1. A method for augmenting a two-stage hand gesture input, comprising:
receiving hand tracking data for a hand of a user;
at a gesture recognition machine, recognizing that the user has performed a first-stage gesture based on one or more parameters derived from the received hand tracking data satisfying first-stage gesture criteria;
providing a visual affordance to the user responsive to recognizing the first-stage gesture, the visual affordance being a graphical interface element rendered so as to appear to float in the space between two digits of the hand of the user without being tethered to either of the two digits, the visual affordance cueing a second-stage gesture, where the second-stage gesture includes the user grasping the visual affordance with the two digits of the hand of the user;
at the gesture recognition machine, recognizing that the user has performed the second-stage gesture based on one or more parameters derived from the received hand tracking data satisfying second-stage gesture criteria; and
responsive to recognizing the second-stage gesture, ceasing to provide the visual affordance and displaying a graphic user interface element in addition to other displayed content.

2. The method of claim 1 , wherein the received hand tracking data includes a feature position for each of a plurality of different hand features at each of a plurality of different frames.

3. The method of claim 2 , wherein the plurality of different hand features include a plurality of palm features, and wherein the first-stage gesture criteria include the plurality of palm features transitioning from facing downwards within a threshold of absolute vertical to facing upwards within a threshold of absolute vertical.

4. The method of claim 3 , wherein the first-stage gesture criteria include the plurality of palm features facing upwards within a threshold of absolute vertical for a threshold duration.

5. The method of claim 2 , wherein the first-stage gesture criteria include a position of the plurality of different hand features within a field of view of the user.

6. The method of claim 2 , wherein the plurality of different hand features include a plurality of fingertip features correlating to an index finger of the hand of the user and a plurality of fingertip features correlating to a thumb of the hand of the user, and wherein the second-stage gesture criteria includes fingertip features correlating to the index finger and fingertip features correlating to the thumb closing to within a threshold distance of the visual affordance.

7. The method of claim 1 , wherein the second-stage gesture criteria include a gaze direction of the user and/or a head orientation of the user.

8. The method of claim 1 , wherein the first-stage gesture criteria include a gaze direction of the user and/or a head orientation of the user.

9. The method of claim 1 , wherein the gesture recognition machine includes an artificial neural network previously trained to recognize the plurality of different hand features.

10. The method of claim 1 , wherein receiving hand tracking data for the hand of the user includes:
receiving depth data for an environment;
fitting a virtual skeleton to point clouds of the received depth data;
assigning hand joints to the virtual skeleton based at least in part on image data of the user performing the first-stage gesture and the second-stage gesture; and
tracking positions of the assigned hand joints across sequential depth images.

11. The method of claim 1 , wherein a visual appearance of the graphical interface element intensifies from a time point when the user initiates the second-stage gesture until the second-stage gesture is recognized.

12. A method for augmenting a two-stage hand gesture input, comprising:
receiving hand tracking data for a hand of a user;
at a gesture recognition machine, recognizing that the user has performed a palm-flip gesture based on one or more parameters derived from the received hand tracking data satisfying palm-flip gesture criteria;
providing a visual affordance to the user responsive to recognizing the palm flip gesture, the visual affordance being a graphical interface element rendered so as to appear adjacent to one or more fingers of the hand of the user, but not rendered so as to be tethered to the one or more fingers of the hand of the user;
at the gesture recognition machine, recognizing that the user has performed a pinch gesture so as to pinch the visual affordance between one of the fingers of the hand of the user and a thumb of the user, based on one or more parameters derived from the received hand tracking data satisfying second-stage gesture criteria; and
responsive to recognizing the pinch gesture, ceasing to provide the visual affordance and displaying a graphic user interface element in addition to other displayed content.

13. The method of claim 12 , wherein the graphic user interface element is displayed in response to recognizing a release of the pinch gesture.

14. The method of claim 13 , where different commands are triggered based on a recognized duration the user holds the pinch gesture prior to releasing the pinch gesture.

15. The method of claim 12 , further comprising:
responsive to recognizing a second pinch gesture while the graphic user interface element is being displayed, closing the graphic user interface element; and
responsive to recognizing a third pinch gesture within a threshold duration of closing the graphic user interface element, reopening the graphic user interface element.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Tang sheng kai
• Schwarz Julia
• Gable Thomas Matthew
• Meekhof Casey Leon
• Qin Chuan
• Sharkasi Nahil Tawfik
• Kamuda Nicholas Ferianc
• Torres Ramiro S
• Neff Joshua Kyle
• Kirschenbaum Jamie Bryant
• Kronlage Neil Richard

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title