Personalized gesture recognition for user interaction with assistant systems

Patent metadata

Field	Value
Publication number	US-11727677-B2
Application number	US-202117566308-A
Country	US
Kind code	B2
Filing date	Dec 30, 2021
Priority date	Apr 20, 2018
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

First claim

What is claimed is:

1. A system comprising:
a microphone configured to receive a speech-input;
a camera configured to receive a gesture-input; and
circuitry configured to:
communicate a user input comprising at least one of the speech-input and the gesture-input to an external assistant system to cause the external assistant system to determine an output user intent from the user input;
in response to communicating the user input, receive information from the external assistant system for the output user intent; and
execute a task based at least in part on the received information for the output user intent determined from at least one of the speech-input and the gesture-input.

2. The system of claim 1 , wherein the circuitry is further configured to execute the task by displaying the received information.

3. The system of claim 1 , wherein the circuitry is further configured to execute the task by outputting audio for the received information.

4. The system of claim 1 , wherein the information is determined at least in part by performing automatic speech recognition on the speech-input.

5. The system of claim 1 , wherein the information is determined at least in part based on the user input and personal information of a user providing the user input.

6. The system of claim 1 , wherein the circuitry is further configured to communicate the user input to the external assistant system via a network.

7. The system of claim 6 , wherein the circuitry is further configured to receive the information from the external assistant system via the network.

8. The system of claim 1 , wherein the circuitry is further configured to determine a modality of the received information from the external assistant system at least in part based on a user profile associated with a user providing the user input.

9. The system of claim 1 , wherein the circuitry is further configured to determine a structure of the received information from the external assistant system at least in part based on a user profile associated with a user providing the user input.

10. The system of claim 1 , wherein the circuitry is further configured to execute the task at least in part based on a user profile associated with a user of the system.

11. The system of claim 10 , wherein the circuitry is further configured to determine the task at least in part based on a machine-learning model that is trained using the user profile.

12. The system of claim 11 , wherein the task comprises recommending an action to the user.

13. The system of claim 1 , wherein the circuitry is further configured to determine an intent of the gesture-input.

14. The system of claim 1 , wherein the user input comprises both the speech-input and the gesture-input.

15. The system of claim 14 , wherein the circuitry is further configured to determine an intent of the gesture-input at least in part based on the speech-input.

16. The system of claim 15 , wherein the circuitry is further configured to determine the intent of the gesture-input using a personalized gesture-classification model associated with a user providing the user input.

17. The system of claim 1 , wherein the microphone and the camera are disposed in augmented reality glasses or a virtual reality headset.

18. The system of claim 1 , wherein the microphone and the camera are enclosed in a computing device corresponding to the circuitry.

19. A method comprising:
receiving a speech-input from a microphone;
receiving a gesture-input from a camera;
communicating a user input comprising at least one of the speech-input and the gesture-input to an external assistant system to cause the external assistant system to determine an output user intent from the user input;
in response to communicating the user input, receiving information from the external assistant system for the output user intent; and
execute a task based at least in part on the received information for the output user intent determined from at least one of the speech-input and the gesture-input.

20. A non-transitory computer-readable medium comprising software that, when executed by a processor, is operable to:
receive a speech-input from a microphone;
receive a gesture-input from a camera;
communicate a user input comprising at least one of the speech-input and the gesture-input to an external assistant system to cause the external assistant system to determine an output user intent from the user input;
in response to communicating the user input, receive information from the external assistant system for the output user intent; and
execute a task based at least in part on the received information for the output user intent determined from at least one of the speech-input and the gesture-input.

Assignees

• Meta Platforms Tech Llc

Inventors

• Liu Xiaohu
• Crook Paul Anthony
• Penov Francislav P
• Subba Rajen

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06Q10/42

  Determination of affinities or common interests between users · CPC title