Hand gesture recognition based on detected wrist muscular movements

Patent metadata

Field	Value
Publication number	US-12073028-B2
Application number	US-202318174358-A
Country	US
Kind code	B2
Filing date	Feb 24, 2023
Priority date	Mar 19, 2021
Publication date	Aug 27, 2024
Grant date	Aug 27, 2024

Abstract

Techniques of identifying gestures include detecting and classifying inner-wrist muscle motions at a user's wrist using micron-resolution radar sensors. For example, a user of an AR system may wear a band around their wrist. When the user makes a gesture to manipulate a virtual object in the AR system as seen in a head-mounted display (HMD), muscles and ligaments in the user's wrist make small movements on the order of 1-3 mm. The band contains a small radar device that has a transmitter and a number of receivers (e.g., three) of electromagnetic (EM) radiation on a chip (e.g., a Soli chip. This radiation reflects off the wrist muscles and ligaments and is received by the receivers on the chip in the band. The received reflected signal, or signal samples, are then sent to processing circuitry for classification to identify the wrist movement as a gesture.

First claim

What is claimed is:

1. A system, comprising:
a wearable device configured to be worn along a portion of an arm, the wearable device including:
a transmitting antenna configured to transmit a pulse of transmitted electromagnetic radiation toward the portion of the arm, and
a receiving antenna configured to receive reflected electromagnetic radiation reflected from the portion of the arm in response to the transmitted electromagnetic radiation being incident on the portion of the arm;

wherein the reflected electromagnetic radiation includes a background component and a foreground component, the background component indicating whether the receiving antenna is activated for determining a gesture, the foreground component indicating the gesture is performed in response to the receiving antenna being activated.

2. The system as in claim 1 , wherein the wearable device is configured to:
perform a classification operation to produce a classification result, the classification result indicating whether the receiving antenna is activated, and the gesture in response to the receiving antenna being activated.

3. The system as in claim 2 , wherein the classification operation further comprises:
inputting values of a representation of the background component into a first convolutional neural network; and
inputting values of a representation of the foreground component into a second convolutional neural network.

4. The system as in claim 3 , wherein the first convolutional neural network includes two convolutional layers and one hidden layer.

5. The system as in claim 2 , wherein the wearable device is further configured to:
in response to the background component indicating that the receiving antenna is not activated, terminating the classification operation.

6. The system as in claim 1 , wherein the receiving antenna configured to receive the reflected electromagnetic radiation reflected from the portion of the arm is further configured to:
obtain sinusoid data from the reflected electromagnetic radiation over channels, wherein the sinusoid data takes a form of waterfall data averaged over the channels.

7. The system as in claim 6 , wherein a sum of the foreground component and the background component is equal to the sinusoid data.

8. A method, comprising:
receiving reflected electromagnetic radiation from a receiving antenna, the receiving antenna being configured to receive the reflected electromagnetic radiation reflected from a wrist in response to transmitted electromagnetic radiation being incident on the wrist from a transmitting antenna and receive electromagnetic radiation reflected from an exterior of the wrist; and
extracting, from the reflected electromagnetic radiation, a background component and a foreground component, the background component indicating whether the receiving antenna is activated for determining a gesture, the foreground component indicating the gesture is performed in response to the receiving antenna being activated.

9. The method as in claim 8 , further comprising:
performing a classification operation to produce a classification result, the classification result indicating whether the receiving antenna is activated, and the gesture in response to the receiving antenna being activated.

10. The method as in claim 9 , wherein performing the classification operation includes:
inputting values of a representation of the background component into a first convolutional neural network; and
inputting values of a representation of the foreground component into a second convolutional neural network.

11. The method as in claim 10 , wherein the first convolutional neural network includes two convolutional layers and one hidden layer.

12. The method as in claim 9 , further comprising:
in response to determining that the receiving antenna is not activated, terminating the classification operation.

13. The method as in claim 8 , wherein receiving the reflected electromagnetic radiation from the receiving antenna includes:
obtaining sinusoid data from the reflected electromagnetic radiation over channels, wherein the sinusoid data takes a form of waterfall data averaged over the channels.

14. The method as in claim 13 , wherein a sum of the foreground component and the background component is equal to the sinusoid data.

15. A computer program product comprising a nontransitive storage medium, the computer program product including code that, when executed by processing circuitry, causes the processing circuitry to perform a method, the method comprising:
receiving reflected electromagnetic radiation from a receiving antenna, the receiving antenna being configured to receive the reflected electromagnetic radiation reflected from a wrist in response to transmitted electromagnetic radiation being incident on the wrist from a transmitting antenna and receive electromagnetic radiation reflected from an exterior of the wrist; and
extracting, from the reflected electromagnetic radiation, a background component and a foreground component, the background component indicating whether the receiving antenna is activated for determining a gesture, the foreground component indicating the gesture is performed in response to the receiving antenna being activated.

16. The computer program product as in claim 15 , wherein the method further comprises:
performing a classification operation to produce a classification result, the classification result indicating whether the receiving antenna is activated, and the gesture in response to the receiving antenna being activated.

17. The computer program product as in claim 16 , wherein performing the classification operation includes:
inputting values of a representation of the background component into a first convolutional neural network; and
inputting values of a representation of the foreground component into a second convolutional neural network.

18. The computer program product as in claim 17 , wherein the first convolutional neural network includes two convolutional layers and one hidden layer.

19. The computer program product as in claim 16 , wherein performing the classification operation further includes:
in response to determining that the receiving antenna is not activated, terminating the classification operation.

20. The computer program product as in claim 15 , wherein receiving the reflected electromagnetic radiation from the receiving antenna includes:
obtaining sinusoid data from the reflected electromagnetic radiation over channels, wherein the sinusoid data takes a form of waterfall data averaged over the channels.

Assignees

• Google Llc

Inventors

• Shin Dongeek
• Izadi Shahram
• Kim David
• Bouaziz Sofien
• Goldberg Steven Benjamin
• Poupyrev Ivan
• Patel Shwetak N

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06F3/014 Primary

  Hand-worn input/output arrangements, e.g. data gloves · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title