Finger gesture recognition via acoustic-optic sensor fusion

Patent metadata

Field	Value
Publication number	US-12468395-B2
Application number	US-202318368358-A
Country	US
Kind code	B2
Filing date	Sep 14, 2023
Priority date	Sep 15, 2022
Publication date	Nov 11, 2025
Grant date	Nov 11, 2025

Abstract

A finger gesture recognition system is provided. The finger gesture recognition system includes one or more audio sensors and one or more optic sensors. The finger gesture recognition system captures, using the one or more audio sensors, audio signal data of a finger gesture being made by a user, and captures, using the one or more optic sensors, optic signal data of the finger gesture. The finger gesture recognition system recognizes the finger gesture based on the audio signal data and the optic signal data and communicates finger gesture data of the recognized finger gesture to an Augmented Reality/Combined Reality/Virtual Reality (XR) application.

First claim

What is claimed is:

1 . A system comprising:
one or more audio sensors; one or more optic sensors; one or more processors operably connected to the one or more audio sensors and the one or more optic sensors; and one or more memories operably connected to the one or more processors, the one or more memories storing instruction that when executed by the one or more processors cause the system to perform operations comprising:
capturing, using the one or more audio sensors, audio signal data of a finger gesture being made by a user;
capturing, using the one or more optic sensors, optic signal data of the finger gesture;
recognizing the finger gesture based on the audio signal data and the optic signal data; and
communicating finger gesture data of the recognized finger gesture to an Extended Reality (XR) application.

2 . The system of claim 1 , wherein the one or more audio sensors are mounted on a volar-wrist portion of a wrist-worn input device.

3 . The system of claim 1 , wherein the one or more optic sensors are mounted on a radial-wrist portion of a wrist-worn input device.

4 . The system of claim 1 , wherein the recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
recognizing the finger gesture using a gesture classification model.

5 . The system of claim 1 , wherein recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
detecting that a thumb of the user is contacting a finger of the user based on the audio signal data using a binary detection model.

6 . The system of claim 5 , wherein recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
determining a swiping finger gesture based on the optic signal data using a thumb tracking component.

7 . The system of claim 1 , wherein recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
aggregating output of a gesture classification model used to recognize the finger gesture using a finite state machine.

8 . A method comprising:
capturing, by one or more processors, using one or more audio sensors, audio signal data of a finger gesture being made by a user; capturing, by the one or more processors, using one or more optic sensors, optic signal data of the finger gesture; recognizing, by the one or more processors, the finger gesture based on the audio signal data and the optic signal data; and communicating, by the one or more processors, finger gesture data of the recognized finger gesture to an XR application.

9 . The method of claim 8 , wherein the one or more audio sensors are mounted on a volar-wrist portion of a wrist-worn input device.

10 . The method of claim 8 , wherein the one or more optic sensors are mounted on a radial-wrist portion of a wrist-worn input device.

11 . The method of claim 8 , wherein the recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
recognizing the finger gesture using a gesture classification model.

12 . The method of claim 8 , wherein recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
detecting that a thumb of the user is contacting a finger of the user based on the audio signal data using a binary detection model.

13 . The method of claim 8 , wherein recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
determining a swiping finger gesture based on the optic signal data using a thumb tracking component.

14 . The method of claim 8 , wherein recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
aggregating output of a gesture classification model used to recognize the finger gesture using a finite state machine.

15 . A computer-storage medium storing instructions that when executed by a computer, cause the computer to perform operations comprising:
capturing, using one or more audio sensors, audio signal data of a finger gesture being made by a user; capturing, using one or more optic sensors, optic signal data of the finger gesture; recognizing the finger gesture based on the audio signal data and the optic signal data; and communicating finger gesture data of the recognized finger gesture to an XR application.

16 . The computer-storage medium of claim 15 , wherein the one or more audio sensors are mounted on a volar-wrist portion of a wrist-worn input device.

17 . The computer-storage medium of claim 15 , wherein the one or more optic sensors are mounted on a radial-wrist portion of a wrist-worn input device.

18 . The computer-storage medium of claim 15 , wherein the recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
recognizing the finger gesture using a gesture classification model.

19 . The computer-storage medium of claim 15 , wherein recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
detecting that a thumb of the user is contacting a finger of the user based on the audio signal data using a binary detection model.

20 . The computer-storage medium of claim 15 , wherein recognizing the finger gesture based on the audio signal data and the optic signal data further comprises:
determining a swiping finger gesture based on the optic signal data using a thumb tracking component.

Assignees

• Snap Inc

Inventors

• Krishnan Gorumkonda Gurunandan
• Nayar Shree K
• Xu Chenhan
• Zhou Bing

Classifications

• G06F3/16

  Sound input; Sound output (speech processing G10L) · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/014

  Hand-worn input/output arrangements, e.g. data gloves · CPC title