Machine-learning based gesture recognition using multiple sensors

Patent metadata

Field	Value
Publication number	US-11449802-B2
Application number	US-202016937481-A
Country	US
Kind code	B2
Filing date	Jul 23, 2020
Priority date	Nov 8, 2019
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

A device implementing a system for machine-learning based gesture recognition includes at least one processor configured to, receive, from a first sensor of the device, first sensor output of a first type, and receive, from a second sensor of the device, second sensor output of a second type that differs from the first type. The at least one processor is further configured to provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type. The at least one processor is further configured to determine the predicted gesture based on an output from the machine learning model, and to perform, in response to determining the predicted gesture, a predetermined action on the device.

First claim

What is claimed is:

1. A method, comprising:
receiving sensor data from a sensor of a device;
providing the sensor data as input to a machine learning model, the machine learning model having been trained to output, while a gesture is being performed by a user of the device and prior to completion of the gesture, a predicted gesture, a predicted start time of the gesture, and a predicted end time of the gesture, based on the sensor data;
determining the predicted gesture based on an output from the machine learning model; and
performing, in response to determining the predicted gesture, a predetermined action on the device.

2. The method of claim 1 , wherein receiving the sensor data comprises receiving the sensor data during a first window of time that at least partially overlaps a gesture time of the gesture, the method further comprising:
receiving additional sensor data from the sensor of the device during a second window of time that at least partially overlaps the gesture time of the gesture;
providing the additional sensor data as input to the machine learning model; and
determining the predicted gesture based on the output from the machine learning model that is based on the sensor data from the first window of time and based on an additional output of the machine learning model that is based on the additional sensor data from the second window of time.

3. The method of claim 2 , wherein determining the predicted gesture based on the output from the machine learning model that is based on the sensor data from the first window of time and based on the additional output of the machine learning model that is based on the additional sensor data from the second window of time comprises:
aggregating a first predicted start time from the machine learning model that is based on the sensor data from the first window of time and a second predicted start time from the machine learning model that is based on the additional sensor data from the second window of time to determine a final predicted start time for the gesture; and
aggregating a first predicted end time from the machine learning model that is based on the sensor data from the first window of time and a second predicted end time from the machine learning model that is based on the additional sensor data from the second window of time to determine a final predicted end time for the gesture.

4. The method of claim 3 , further comprising adjusting a size of an input buffer for the machine learning model based on the final predicted start time and the final predicted end time.

5. The method of claim 4 , wherein determining the predicted gesture comprises determining the predicted gesture at a time after the final predicted end time using sensor data in the input buffer having the adjusted size.

6. The method of claim 1 , the machine learning model having been trained to output the predicted start time of the gesture and the predicted end time of the gesture at least in part by generating a gesture label and a no-gesture label for each of multiple windows of time.

7. The method of claim 6 , wherein the gesture label and the no-gesture label each have a value that is smoothly continuous between a maximum value and a minimum value.

8. The method of claim 1 , wherein the predetermined action corresponds to changing a user interface on the device.

9. The method of claim 1 , wherein the predetermined action comprises sending gesture information or an instruction from the device to a companion device.

10. A device, comprising:
at least one processor; and
a memory including instructions that, when executed by the at least one processor, cause the at least one processor to:
receive sensor data from a sensor of the device;
provide the sensor data as input to a machine learning model, the machine learning model having been trained to output, while a gesture is being performed by a user of the device and prior to completion of the gesture, a predicted gesture, a predicted start time of the gesture, and a predicted end time of the gesture, based on the sensor data;
determine the predicted gesture based on an output from the machine learning model; and
perform, in response to determining the predicted gesture, a predetermined action on the device.

11. The device of claim 10 , wherein the one or more processors are configured to receive the sensor data during a first window of time that at least partially overlaps a gesture time of the gesture, and are further configured to:
receive additional sensor data from the sensor of the device during a second window of time that at least partially overlaps the gesture time of the gesture;
provide the additional sensor data as input to the machine learning model; and
determine the predicted gesture based on the output from the machine learning model that is based on the sensor data from the first window of time and based on an additional output of the machine learning model that is based on the additional sensor data from the second window of time.

12. The device of claim 11 , wherein the one or more processors are configured to determine the predicted gesture based on the output from the machine learning model that is based on the sensor data from the first window of time and based on the additional output of the machine learning model that is based on the additional sensor data from the second window of time by:
aggregating a first predicted start time from the machine learning model that is based on the sensor data from the first window of time and a second predicted start time from the machine learning model that is based on the additional sensor data from the second window of time to determine a final predicted start time for the gesture; and
aggregating a first predicted end time from the machine learning model that is based on the sensor data from the first window of time and a second predicted end time from the machine learning model that is based on the additional sensor data from the second window of time to determine a final predicted end time for the gesture.

13. The device of claim 12 , wherein the one or more processors are further configured to adjust a size of an input buffer for the machine learning model based on the final predicted start time and the final predicted end time.

14. The device of claim 13 , wherein the one or more processors are configured to determine the predicted gesture at a time after the final predicted end time, using sensor data in the input buffer having the adjusted size.

15. The device of claim 10 , the machine learning model having been trained to output the predicted start time of the gesture and the predicted end time of the gesture at least in part by generating a gesture label and a no-gesture label for each of multiple windows of time.

16. The device of claim 15 , wherein the gesture label and the no-gesture label each have a value that is smoothly continuous between a maximum value and a minimum value.

17. The device of claim 10 , further comprising the sensor.

18. The device of claim 17 , wherein the device comprises a smartwatch.

19. The device of claim 18 , wherein the at least one processor is configured to receive the sensor data from the sensor of the device while the smartwatch is worn on a wrist of the user, and wherein the gesture corresponds to a single-handed gesture performed by a hand that is coupled to the wrist on which the smartwatch is worn.

20. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors,

Assignees

• Apple Inc

Inventors

• Maalouf Charles
• Scully Shawn R
• Fleizach Christopher B
• Nguyen Tu K
• Liang Lilian H
• Seto Warren J
• Quintana Julian
• Beyhs Michael J
• Seyed Mousavi Hojjat
• Shahsavari Behrooz

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06F2218/12

  Classification; Matching · CPC title

• G06N3/045

  Combinations of networks · CPC title