Methods and apparatuses for identifying gestures based on ultrasound data

Patent metadata

Field	Value
Publication number	US-11048334-B2
Application number	US-201816229765-A
Country	US
Kind code	B2
Filing date	Dec 21, 2018
Priority date	Dec 22, 2017
Publication date	Jun 29, 2021
Grant date	Jun 29, 2021

Abstract

Aspects of the technology described herein relate to methods and apparatuses for identifying gestures based on ultrasound data. Performing gesture recognition may include obtaining, with a wearable device, ultrasound data corresponding to an anatomical gesture; and identifying the anatomical gesture based on the obtained ultrasound data. Interfacing with a computing device may include identifying, with a wearable device, an anatomical gesture using ultrasound data obtained by the wearable device; and causing the computing device to perform a specific function based on the anatomical gesture identified by the wearable device. Training a wearable device to perform gesture recognition may include obtaining, with the wearable device, ultrasound data corresponding to an anatomical gesture; obtaining non-ultrasound data corresponding to the anatomical gesture; and training a machine learning model accessed by the wearable device to recognize the anatomical gesture based on correlating the non-ultrasound data and the ultrasound data.

First claim

What is claimed is:

1. A system for training a wearable device to perform gesture recognition based on ultrasound data, comprising:
a memory circuitry;
a processing circuitry in communication with the memory circuitry, the processing circuitry configured to:
obtain, from the wearable device, ultrasound data generated at a first time in response to performance of an anatomical gesture;
obtain an image of the anatomical gesture from an image capture device on a smartphone or a tablet device separate from the wearable device; and
train, by correlating the image of the anatomical gesture with the ultrasound data generated at the first time in response to performance of the anatomical gesture, a machine learning model accessed by the wearable device to recognize the anatomical gesture from an input of ultrasound data collected at a second time and to output a label of the anatomical gesture in response to the input of ultrasound data collected at the second time.

2. The system of claim 1 , wherein the wearable device comprises one of a wristwatch or a wristband configured with an ultrasound-on-a-chip.

3. The system of claim 2 , wherein the wearable device further comprises one or more position sensors.

4. The system of claim 3 , wherein the one or more position sensors comprise one or more accelerometers, gyroscopes, magnetometers, compasses, and global positioning systems (GPS).

5. The system of claim 3 , wherein the processing circuitry is further configured to obtain position data from the one or more position sensors and to train the machine learning model accessed by the wearable device to recognize the anatomical gesture by correlating the position data with the image of the anatomical gesture and the ultrasound data generated at the first time in response to performance of the anatomical gesture.

6. The system of claim 1 , wherein the anatomical gesture comprises one of a hand gesture, a finger gesture, a wrist gesture and/or an arm gesture.

7. A method to train a wearable device to perform gesture recognition based on ultrasound data, comprising:
obtaining, with the wearable device, ultrasound data generated at a first time in response to performance of an anatomical gesture;
obtaining an image of the anatomical gesture from an image capture device on a smartphone or a tablet device separate from the wearable device; and
training, by correlating the image of the anatomical gesture with the ultrasound data generated at the first time in response to performance of the anatomical gesture, a machine learning model accessible to the wearable device to recognize the anatomical gesture from an input of ultrasound data collected at a second time and to output a label of the anatomical gesture in response to the input of the ultrasound data collected at the second time.

8. The method of claim 7 , wherein the wearable device comprises one of a wristwatch or a wristband.

9. The method of claim 8 , wherein the wearable device further comprises an ultrasound-on-a-chip device.

10. The method of claim 9 , wherein the wearable device further comprises one or more position sensors.

11. The method of claim 10 , wherein the method further comprises obtaining position data from the one or more position sensors, and wherein training the machine learning model accessible to the wearable device to recognize the anatomical gesture comprises training the machine learning model to recognize the anatomical gesture by correlating the position data with the image of the anatomical gesture and the ultrasound data generated at the first time in response to performance of the anatomical gesture.

12. The method of claim 11 , wherein the one or more position sensors comprise one or more accelerometers, gyroscopes, magnetometers, compasses, and global positioning systems (GPS).

13. The method of claim 7 , wherein the anatomical gesture comprises one of a hand gesture, a finger gesture, a wrist gesture and/or an arm gesture.

14. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to:
obtain, from a wearable device, ultrasound data generated at a first time in response to performance of an anatomical gesture;
obtain an image of the anatomical gesture from an image capture device on a smartphone or a tablet device separate from the wearable device; and
train, by correlating the image of the anatomical gesture with the ultrasound data generated at the first time in response to performance of the anatomical gesture, a machine learning model accessed by the wearable device to recognize the anatomical gesture from an input of ultrasound data collected at a second time and to output a label of the anatomical gesture in response to the input of the ultrasound data collected at the second time.

15. The at least one non-transitory computer-readable storage medium of claim 14 , wherein the wearable device comprises one of a wristwatch or a wristband.

16. The at least one non-transitory computer-readable storage medium of claim 15 , wherein the wearable device further comprises an ultrasound-on-a-chip device.

17. The at least one non-transitory computer-readable storage medium of claim 16 , wherein the wearable device further comprises one or more position sensors.

18. The at least one non-transitory computer-readable storage medium of claim 17 , wherein the one or more position sensors comprise one or more accelerometers, gyroscopes, magnetometers, compasses, and global positioning systems (GPS).

19. The at least one non-transitory computer-readable storage medium of claim 17 , wherein the at least one non-transitory computer readable storage medium further stores processor-executable instructions that, when executed by the at least one processor, cause the at least one processor to obtain position data using the one or more position sensors, and to train the machine learning model accessed by the wearable device to recognize the anatomical gesture by correlating the position data with the image of the anatomical gesture and the ultrasound data generated at the first time in response to performance of the anatomical gesture.

20. The at least one non-transitory computer-readable storage medium of claim 14 , wherein the anatomical gesture comprises one of a hand gesture, a finger gesture, a wrist gesture and/or an arm gesture.

Assignees

• Butterfly Network Inc

Inventors

• Rothberg Jonathan M
• Ralston Tyler S
• Silberman Nathan

Classifications

• G06V10/774 Primary

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/143

  Sensing or illuminating at different wavelengths · CPC title