Trainable sensor-based gesture recognition

Patent metadata

Field	Value
Publication number	US-9405377-B2
Application number	US-201414214707-A
Country	US
Kind code	B2
Filing date	Mar 15, 2014
Priority date	Mar 15, 2014
Publication date	Aug 2, 2016
Grant date	Aug 2, 2016

Abstract

In many computing scenarios, a device comprises at least one sensor, and is configured to recognize a gesture performed by a user according to the sensor output of the sensor, and to perform a particular action upon recognizing the gesture. However, many devices are preconfigured with such gestures, and the recognition is specific to the sensors of the device, and is not specific to the manner in which a particular user performs the gesture. Presented herein are techniques for enabling a device to recognize a new gesture by monitoring the sensor output of any sensors provided by the device while the user performs the gesture, optionally requesting repeated gesture performances until reaching a recognition confidence. Once trained to recognize the gesture according to the sensor outputs of the particular sensors of the device, the device may subsequently recognize the gesture performed by the user and execute an associated action.

First claim

What is claimed is:

1. A method of performing an action associated with a gesture performed by a user of a device having a processor and a sensor providing a sensor output, the method comprising:
executing on the processor instructions that cause the device to:
transition the device to a training mode upon receiving, from the user, a training start, request;
during the training mode:
while the user performs the gesture, monitor the sensor to detect an identified sensor output identifying the gesture during a training period between the training start request and the training completion request;
identify, during the training period, a restricted period that is shorter than the training period and during which the identified sensor output identifies the gesture;
associate with the gesture the identified sensor output of the sensor during the restricted period; and
associate an action with the gesture;

transition the device to a recognition mode upon receiving, from the user, a training completion request; and
during the recognition mode:
monitor the sensor output of the sensor; and
upon detecting that the sensor output matches the identified sensor output associated with the gesture, perform the action associated with the gesture.

2. The method of claim 1 , wherein executing the instructions on the processor further causes the device to, upon detecting an addition to the device of an added sensor, transition the device to the training mode to detect the identified sensor output of the added sensor while the user performs the gesture.

3. The method of claim 1 , wherein executing the instructions on the processor further causes the device to, upon detecting a gesture that is not associated with an identified sensor output of a sensor within a recognition confidence threshold, transition the device to the training mode to retrain the sensor to recognize the gesture.

4. The method of claim 1 , wherein executing the instructions on the processor further causes the device to, upon detecting a miscalibration of a sensor that reduces a recognition accuracy of the identified sensor output associated with the gesture, transition the device to the training mode to retrain the sensor to recognize the gesture.

5. The method of claim 1 , wherein executing the instructions on the processor further causes the device to, upon receiving a gesture profile specifying a selected gesture that is associated with an identified sensor output that is not compatible with the sensor of the device, transition the device to the training mode to train the sensor to recognize the gesture.

6. The method of claim 1 , wherein executing the instructions on the processor further causes the device to, upon detecting a sensor output of the sensor identifying a detected gesture that is not associated with an action, request the user to associate the detected gesture with at least one action.

7. The method of claim 1 , wherein monitoring the sensor to detect the identified sensor output further comprises:
during respective at least two gesture performances of the gesture by the user, monitoring the sensor to detect the identified sensor output identifying the gesture during the gesture performance; and
determining the identified sensor output according to the identified sensor outputs of the respective at least two gesture performances.

8. The method of claim 7 , wherein determining the identified sensor output further comprises:
comparing the identified sensor outputs during the respective at least two gesture performances to determine a recognition confidence;
while the recognition confidence is above a recognition confidence threshold, instructing the user to repeat the gesture performance; and
upon determining the recognition confidence within the recognition confidence threshold, associating the identified sensor output with the gesture.

9. The method of claim 1 , wherein the association between the identified sensor output and the gesture is stored in a gesture profile, the method further comprising executing on the processor instructions that cause the device to send the gesture profile to a gesture profile service.

10. The method of claim 1 , further comprising executing on the processor instructions that cause the device to retrieve a gesture profile from a gesture profile service, the gesture profile comprising an association of a gesture with an identified sensor output of at least one sensor and an action.

11. The method of claim 10 , further comprising executing on the processor instructions that cause the device to transition the device to the training mode in response to detecting that the gesture profile's identified sensor output of the at least one sensor comprises two or more sensor outputs of two or more sensors, at least one of which is absent from the device.

12. The method of claim 10 , further comprising executing on the processor instructions that cause the device to:
detect that the gesture profile's identified sensor output of the at least one sensor comprises two or more sensor outputs of two or more sensors, at least one of which is absent from the device; and
translate the at least one sensor output of the at least one sensor that is absent from the device into translated sensor output of the sensor that is part of the device.

13. A device that performs actions associated with gestures performed by a user, the device comprising:
a memory;
at least two sensors providing at least two sensor outputs;
a gesture trainer that:
while the user performs the gesture, monitors the at least two sensors to detect one or more identified sensor outputs identifying the gesture, the detecting the one or more identified sensor output comprises:
identifying, at least one recognizing sensor providing an identified sensor output identifying the gesture; and
identifying at least one non-recognizing sensor that does not provide an identified sensor output identifying the gesture; and

stores in the memory the one or more identified sensor outputs that together identify the gesture, the storing excluding, the identified sensor output of the at least one non-recognizing sensor;

an action associator that, upon receiving from the user an action to be associated with the gesture, stores in the memory an association of the action with the gesture; and
a gesture recognizer that:
monitors the at least two sensor outputs of the at least two sensors to detect the one or more identified sensor outputs identifying the gesture; and
performs the action associated in the memory with the gesture associated with the one or more identified sensor outputs.

14. The device of claim 13 , wherein:
detecting the identified sensor output identifying the gesture further comprises:
requesting the user to identify, among the at least two sensors, at least one selected sensor identifying the gesture;
while the user performs the gesture, monitoring only the at least one selected sensor to detect an identified sensor output identifying the gesture; and

storing the one or more identified sensor outputs in the memory further comprises: storing in the memory the identified sensor outputs only of the respective at least one selected sensor.

15. The device of claim 14 , wherein:
the respective devices are associated with at least one sensor modality;
requesting the user to identify the at least one selected sensor further comprises:
requesting the user to identify, among at least two sensor modalities respectively associated with at least one sensor of the device, at least one selected sensor modality identifying the gesture;
detecting

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Li Mengke

Classifications

• G01P15/00

  Measuring acceleration; Measuring deceleration; Measuring shock, i.e. sudden change of acceleration · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G01H17/00

  Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves, not provided for in the other groups of this subclass · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title