In-call experience enhancement for assistant systems

Patent metadata

Field	Value
Publication number	US-2025037212-A1
Application number	US-202418883896-A
Country	US
Kind code	A1
Filing date	Sep 12, 2024
Priority date	Oct 18, 2019
Publication date	Jan 30, 2025
Grant date	—

Abstract

In one embodiment, a method includes establishing a video call between a plurality of client systems, wherein access to an assistant system is persistently maintained during the video call, receiving, from a first client system of the plurality of client systems, a request by a first user to be performed by the assistant system during the video call, wherein the request references one or more activities associated with one or more users associated with the plurality of client systems, analyzing, by a context engine of the assistant system, images of a scene of the video call to identify the one or more activities within the scene, instructing the assistant system to execute the request based on the identified one or more activities, and sending, to one or more of the plurality of client systems, a response to the request while maintaining the video call between the plurality of client systems.

First claim

1 . (canceled)

2 . A method comprising, by one or more computing systems,
establishing a video call between a plurality of client systems, wherein the video call comprises a scene; invoking an assistant system during the video call; analyzing images of the scene of the video call to identify one or more objects within the scene; receiving, from a first client system of the plurality of client systems, a request to be performed by the assistant system during the video call, wherein the request references a specific object of the one or more objects; executing, by the assistant system, the request based at least in part on the specific object; and providing, to one or more of the plurality of client systems, a response to the request.

3 . The method of claim 2 , wherein the request is a voice request made by a user of the first client system.

4 . The method of claim 2 , wherein executing the request comprises adjusting a camera associated with the video call based at least in part on a location of the specific object within the scene.

5 . The method of claim 2 , further comprising:
storing relationship data comprising a relationship between the specific object and a user of the plurality of client systems, wherein the request is executed based at least in part on the relationship data.

6 . The method of claim 2 , wherein access to the assistant system is persistently maintained during the video call.

7 . The method of claim 2 , wherein the response to the request is provided while maintaining the video call between the plurality of client systems.

8 . A non-transitory, non-volatile computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to:
establish a video call between a plurality of client systems, wherein the video call comprises a scene; invoke an assistant system during the video call; analyze images of the scene of the video call to identify one or more objects within the scene; receive, from a first client system of the plurality of client systems, a request to be performed by the assistant system during the video call, wherein the request references a specific object of the one or more objects; execute, by the assistant system, the request based at least in part on the specific object; and provide, to one or more of the plurality of client systems, a response to the request.

9 . The non-transitory, non-volatile computer-readable medium of claim 8 , wherein the request is a voice request made by a user of the first client system.

10 . The non-transitory, non-volatile computer-readable medium of claim 8 , wherein executing the request comprises adjusting a camera associated with the video call based at least in part on a location of the specific object within the scene.

11 . The non-transitory, non-volatile computer-readable medium of claim 8 , wherein the instructions, when executed by one or more processors of the computing system, further cause the computing system to:
store relationship data comprising a relationship between the specific object and a user of the plurality of client systems, wherein the request is executed based at least in part on the relationship data.

12 . The non-transitory, non-volatile computer-readable medium of claim 8 , wherein access to the assistant system is persistently maintained during the video call.

13 . The non-transitory, non-volatile computer-readable medium of claim 8 , wherein the response to the request is provided while maintaining the video call between the plurality of client systems.

14 . A client system comprising:
one or more processors; a non-transitory computer-readable media; a camera configured to capture a video comprising a scene containing one or more objects; a communication interface configured to transmit the video over a network in real-time; and a microphone configured to receive a voice request from a user, the voice request referencing a specific object of the one or more objects in the scene, wherein the client system is configured to invoke an assistant system during capture of the video and to provide information about the voice request, including information about the specific object, to the assistant system, and wherein the client system is configured to receive a response to the voice request from the assistant system.

15 . The client system of claim 14 , wherein at least a portion of the assistant system is remotely connected to the client system via the network.

16 . The client system of claim 14 , wherein the video is part of a video call with at least one other client system connected to the client system via the network.

17 . The client system of claim 16 , wherein access to the assistant system is persistently maintained during the video call.

18 . The client system of claim 16 , wherein the response to the voice request is received while maintaining the video call.

19 . The client system of claim 14 , wherein executing the voice request comprises adjusting a camera associated with the video based at least in part on a location of the specific object within the scene.

20 . The client system of claim 14 , wherein the client system is further configured to:
store relationship data comprising a relationship between the specific object and the user of the client system, wherein the voice request is executed based at least in part on the relationship data.

Assignees

• Meta Platforms Tech Llc

Inventors

• Liu Xiaohu
• Liu Baiyang
• Subba Rajen

Classifications

• G06Q10/40 Primary

  Business processes related to social networking or social networking services · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06Q10/1093

  Calendar-based scheduling for persons or groups · CPC title

• G10L2015/0631

  Creating reference templates; Clustering · CPC title