Gesture recognition

Patent metadata

Field	Value
Publication number	US-10488939-B2
Application number	US-201715671118-A
Country	US
Kind code	B2
Filing date	Aug 7, 2017
Priority date	Apr 20, 2017
Publication date	Nov 26, 2019
Grant date	Nov 26, 2019

Abstract

A gesture recognition method comprises receiving at a processor from a sensor a sequence of captured signal frames for extracting hand pose information for a hand and using at least one trained predictor executed on the processor to extract hand pose information from the received signal frames. For at least one defined gesture, defined as a time sequence comprising hand poses, with each of the hand poses defined as a conjunction or disjunction of qualitative propositions relating to interest points on the hand, truth values are computed for the qualitative propositions using the hand pose information extracted from the received signal frames, and execution of the gesture is tracked, by using the truth values to determine which of the hand poses in the time sequence have already been executed and which of the hand poses in the time sequence is expected next.

First claim

What is claimed is:

1. A gesture recognition method comprising:
receiving at a processor from a sensor a sequence of captured signal frames for extracting hand pose information for a hand;
using at least one trained predictor executed on the processor to extract the hand pose information from the received signal frames;
for at least one defined gesture, wherein the gesture is defined as a time sequence comprising hand poses, wherein each of the hand poses is defined as a conjunction or disjunction of qualitative propositions relating to interest points on the hand:
computing truth values for the qualitative propositions using the hand pose information extracted from the received signal frames, and
tracking execution of the gesture, by using the truth values to determine which of the hand poses in the time sequence have already been executed and which of the hand poses in the time sequence is expected next, wherein upon completion of the sequence, a function associated with the gesture is triggered.

2. A gesture recognition method according to claim 1 , wherein the interest points comprise the fingertips and palm center of the hand.

3. A gesture recognition method according to claim 1 , wherein the qualitative propositions are obtained by applying predicates to the interest points of the hand.

4. A gesture recognition method according to claim 3 , wherein the predicates comprise at least one of the following predicates: palm direction, palm orientation, finger direction, finger flexion, finger tangency, finger relative position.

5. A gesture recognition method according to claim 1 , wherein the hand pose information is extracted in multiple stages, wherein at each of the stages a piece of the hand pose information is extracted using a set of one or more predictors.

6. A gesture recognition method according to claim 5 , wherein the piece of hand pose information is extracted from each of the signal frames at a later one of the stages in dependence on the piece of hand pose information extracted from that signal frame at an earlier one of the stages.

7. A gesture recognition method according to claim 6 , wherein only a subset of one or more predictors selected from a set of available trained predictors is activated at the later stage to extract the piece of hand pose information from that signal frame at the later stage, that subset of predictors being selected for that signal frame based on the piece of information extracted from that signal frame at the earlier stage.

8. A gesture recognition method according to claim 7 , wherein at the earlier stage a trained classifier is used to classify the signal frame as belonging to at least one of a plurality of global hand orientation clusters, the piece of hand pose information being extracted at the later stage in dependence on the global hand orientation cluster to which the signal frame belongs.

9. A gesture recognition method according to claim 8 , wherein at the later stage the piece of hand pose information is extracted using at least one trained regressor.

10. A gesture recognition method according to claim 9 , wherein only a subset of one or more regressors selected from a set of available trained regressors is activated at the later stage to extract the piece of hand pose information from that signal frame at the later stage, that subset of regressors being selected for that signal frame based on the global hand orientation cluster to which the signal frame belongs.

11. A gesture recognition method according to claim 10 , wherein the subset of one or more regressors is used to determine a refined global hand orientation estimate, or location information for at least one finger of the hand.

12. A gesture recognition method according to claim 11 , wherein regression is performed separately for multiple fingers to determine location information for each of those fingers.

13. A gesture recognition method according to claim 9 , wherein a fixed number of regression stages is performed for each of the signal frames.

14. A gesture recognition method according to claim 1 , wherein the predictors are convolutional tables ensemble (CTE) predictors.

15. A gesture recognition method according to claim 1 , wherein extracting the hand pose information comprises estimating a hand pose for each of the signal frames.

16. A gesture recognition method according to claim 1 , wherein the signal frames are filtered signal frames generated by applying a temporal filter to unfiltered frames captured by the sensor, whereby each of the filtered signal frames comprises information from multiple unfiltered frames.

17. A method according to claim 1 , wherein the sensor comprises a camera and the signal frames are images of the hand captured by the camera.

18. A gesture recognition method according to claim 1 , wherein the time sequence also comprises at least one motion element, which is a qualitative indicator of hand motion.

19. A storage device storing executable instructions that, when executed on a processor, cause the processor to implement a method comprising:
receiving at the processor from a sensor a sequence of captured signal frames for extracting hand pose information for a hand;
using at least one trained predictor executed on the processor to extract hand pose information from the received signal frames;
for at least one defined gesture, wherein the gesture is defined as a time sequence comprising hand poses, wherein each of the hand poses is defined as a conjunction or disjunction of qualitative propositions relating to interest points on the hand:
computing truth values for the qualitative propositions using the hand pose information extracted from the received signal frames, and
tracking execution of the gesture, by using the truth values to determine which of the hand poses in the time sequence have already been executed and which of the hand poses in the time sequence is expected next, wherein upon completion of the sequence, a function associated with the gesture is triggered.

20. A gesture recognition device comprising:
a sensor for use in capturing a sequence of captured signal frames for extracting hand pose information for a hand;
a processor configured to receive from the sensor a sequence of captured signal frames for extracting hand pose information for a hand, and to execute at least one trained predictor to extract hand pose information from the received signal frames;
wherein the processor is configured to implement the following operations for at least one defined gesture, wherein the gesture is defined as a time sequence comprising hand poses, wherein each of the hand poses is defined as a conjunction or disjunction of qualitative propositions relating to interest points on the hand:
computing truth values for the qualitative propositions using the hand pose information extracted from the received signal frames, and
tracking execution of the gesture, by using the truth values to determine which of the hand poses in the time sequence have already been executed and which of the hand poses in the time sequence is expected next, wherein upon completion of the sequence, the processor is configured to trigger a function associated with the gesture.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Karmon Kfir
• Bar-Hillel Aharon
• Krupka Eyal
• Bloom Noam
• Gurvich Ilya
• Hurvitz Aviv
• Leichter Ido
• Smolin Yoni
• Tzairi Yuval
• Vinnikov Alon

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06K9/00355

  Physics · mapped topic

• G06K9/00389

  Physics · mapped topic

• G06K9/4604

  Physics · mapped topic