Personalized gesture recognition for user interaction with assistant systems

Patent metadata

Field	Value
Publication number	US-12475698-B2
Application number	US-202418915864-A
Country	US
Kind code	B2
Filing date	Oct 15, 2024
Priority date	Apr 20, 2018
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

First claim

What is claimed is:

1 . A method comprising, by a computer system:
receiving a first gesture-input of a user and a speech-input of the user; determining a user intent based at least in part on the speech-input; determining, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input; associating the gesture identifier with the user intent; receiving a second gesture-input of the user; determining if the second gesture-input corresponds to the gesture identifier; and in response to determining that the second gesture-input corresponds to the gesture identifier, performing a task based at least in part on the user intent associated with the gesture identifier.

2 . The method according to claim 1 , wherein the gesture identifier corresponds to one of a pinch gesture, a swipe gesture, or an open palm gesture.

3 . The method according to claim 1 , wherein the computer system comprises smart glasses.

4 . The method according to claim 1 , wherein an assistant system performs the task, and the task comprises presenting a response to the user via the computer system.

5 . The method according to claim 4 , wherein the response comprises at least one of text, an image, a video, or an animation of a gesture.

6 . The method according to claim 4 , wherein the assistant system is external to the computer system and communicatively coupled to the computer system via a network.

7 . The method according to claim 4 , wherein the assistant system comprises an artificial neural network.

8 . The method according to claim 1 , wherein:
the first gesture-input comprises an arc motion by a hand of the user with at least two fingers extended, the user intent corresponds to a next instruction, and the gesture identifier is determined to be a swipe gesture.

9 . A non-transitory, computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to:
receive a first gesture-input of a user and a speech-input of the user; determine a user intent based at least in part on the speech-input; determine, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input; associate the gesture identifier with the user intent; receive a second gesture-input of the user; determine if the second gesture-input corresponds to the gesture identifier; and in response to determining that the second gesture-input corresponds to the gesture identifier, perform a task based at least in part on the user intent associated with the gesture identifier.

10 . The non-transitory, computer-readable medium according to claim 9 , wherein the gesture identifier corresponds to one of a pinch gesture, a swipe gesture, or an open palm gesture.

11 . The non-transitory, computer-readable medium according to claim 9 , wherein the computer system comprises smart glasses.

12 . The non-transitory, computer-readable medium according to claim 9 , wherein an assistant system performs the task, and the task comprises presenting a response to the user via the computer system.

13 . The non-transitory, computer-readable medium according to claim 12 , wherein the response comprises at least one of text, an image, a video, or an animation of a gesture.

14 . The non-transitory, computer-readable medium according to claim 12 , wherein the assistant system is external to the computer system and communicatively coupled to the computer system via a network.

15 . The non-transitory, computer-readable medium according to claim 12 , wherein the assistant system comprises an artificial neural network.

16 . The non-transitory, computer-readable medium according to claim 9 , wherein:
the first gesture-input comprises an arc motion by a hand of the user with at least two fingers extended, the user intent corresponds to a next instruction, and the gesture identifier is determined to be a swipe gesture.

17 . A smart glasses system, comprising:
a microphone; a camera; one or more processors communicatively coupled to the microphone and the camera; and one or more non-transitory, computer-readable media storing instructions that, when executed by at least one of the one or more processors, cause the smart glasses system to:
receive from the camera a first gesture-input of a user and from the microphone a speech-input of the user;
determine a user intent based at least in part on the speech-input;
determine, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input;
associate the gesture identifier with the user intent;
receive from the camera a second gesture-input of the user;
determine if the second gesture-input corresponds to the gesture identifier; and
in response to determining that the second gesture-input corresponds to the gesture identifier, perform a task based at least in part on the user intent associated with the gesture identifier.

18 . The smart glasses system according to claim 17 , wherein an assistant system performs the task, and the task comprises presenting a response to the user via the smart glasses system.

19 . The smart glasses system according to claim 18 ,
wherein the response comprises at least one of text, an image, a video, or an animation of a gesture, and wherein the gesture identifier corresponds to one of a pinch gesture, a swipe gesture, or an open palm gesture.

20 . The smart glasses system according to claim 17 , wherein:
the first gesture-input comprises an arc motion by a hand of the user with at least two fingers extended, the user intent corresponds to a next instruction, and the gesture identifier is determined to be a swipe gesture.

Assignees

• Meta Platforms Tech Llc

Inventors

• Crook Paul Anthony
• Liu Xiaohu
• Penov Francislav P
• Subba Rajen

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06F21/6245

  Protecting personal data, e.g. for financial or medical purposes · CPC title

• H04L5/02

  Channels characterised by the type of signal · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title