Machine-learning based gesture recognition using multiple sensors

Patent metadata

Field	Value
Publication number	US-11699104-B2
Application number	US-202217869740-A
Country	US
Kind code	B2
Filing date	Jul 20, 2022
Priority date	Nov 8, 2019
Publication date	Jul 11, 2023
Grant date	Jul 11, 2023

Abstract

A device implementing a system for machine-learning based gesture recognition includes at least one processor configured to, receive, from a first sensor of the device, first sensor output of a first type, and receive, from a second sensor of the device, second sensor output of a second type that differs from the first type. The at least one processor is further configured to provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type. The at least one processor is further configured to determine the predicted gesture based on an output from the machine learning model, and to perform, in response to determining the predicted gesture, a predetermined action on the device.

First claim

What is claimed is:

1. A method comprising:
receiving, from a first sensor of a device, first sensor output of a first type;
receiving, from a second sensor of the device, second sensor output of a second type that differs from the first type;
providing the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type;
determining the predicted gesture based on an output from the machine learning model; and
performing, in response to determining the predicted gesture, a predetermined action on the device.

2. The method of claim 1 , wherein the first sensor of the device comprises a photoplethysmography (PPG) sensor.

3. The method of claim 2 , wherein the PPG sensor comprises at least one of an infrared light source or a color light source.

4. The method of claim 2 , wherein the first sensor output indicates a change in blood flow.

5. The method of claim 1 , wherein the second sensor of the device comprises at least one of an accelerometer or a microphone.

6. The method of claim 1 , wherein at least one of receiving the first sensor output or receiving the second sensor output is based on a determination that the device is in a gesture detection mode.

7. The method of claim 1 , the machine learning model having been trained across multiple different users.

8. The method of claim 1 , wherein the predicted gesture comprises at least one of a finger-based gesture, or a wrist-based gesture.

9. The method of claim 8 , wherein the finger-based gesture comprises at least one of a finger pinch gesture or a fist-clinch gesture.

10. The method of claim 1 , wherein the predetermined action corresponds to changing a user interface on the device.

11. A device, comprising:
a first sensor;
a second sensor;
at least one processor; and
a memory including instructions that, when executed by the at least one processor, cause the at least one processor to:
receive, from the first sensor, first sensor output of a first type;
receive, from the second sensor, second sensor output of a second type that differs from the first type;
provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type;
determine the predicted gesture based on an output from the machine learning model; and
perform, in response to determining the predicted gesture, a predetermined action on the device.

12. The device of claim 11 , wherein the first sensor comprises a photoplethysmography (PPG) sensor.

13. The device of claim 12 , wherein the PPG sensor comprises at least one of an infrared light source or a color light source.

14. The device of claim 12 , wherein the first sensor output indicates a change in blood flow.

15. The device of claim 11 , wherein the second sensor the device comprises at least one of an accelerometer or a microphone.

16. The device of claim 11 , wherein at least one of receiving the first sensor output or receiving the second sensor output is based on a determination that the device is in a gesture detection mode.

17. The device of claim 11 , the machine learning model having been trained across a general population of users.

18. The device of claim 11 , wherein the predicted gesture comprises at least one of a finger-based gesture, or a wrist-based gesture.

19. The device of claim 18 , wherein the finger-based gesture comprises at least one of a finger pinch gesture or a fist-clinch gesture.

20. A computer program product comprising code, stored in a non-transitory computer-readable storage medium, the code comprising:
code to receive, from a first sensor of a device, first sensor output of a first type, the first sensor comprising a bio-signal sensor;
code to receive, from a second sensor of the device, second sensor output of a second type that differs from the first type;
code to provide the first sensor output and the second sensor output as inputs to a machine learning model, the machine learning model having been trained to output a predicted gesture based on sensor output of the first type and sensor output of the second type;
code to determine the predicted gesture based on an output from the machine learning model; and
code to perform, in response to determining the predicted gesture, a predetermined action on the device.

21. The computer program product of claim 20 , the code further comprising:
code to perform a gesture registration processes for a user of the device; and
code to tune the machine learning model for the user based on the gesture registration process.

Assignees

• Apple Inc

Inventors

• Maalouf Charles
• Scully Shawn R
• Fleizach Christopher B
• Nguyen Tu K
• Liang Lilian H
• Seto Warren J
• Quintana Julian
• Beyhs Michael J
• Seyed Mousavi Hojjat
• Shahsavari Behrooz

Classifications

• G06F3/015

  Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06N3/08

  Learning methods · CPC title

• G06F3/04883

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title