Assistance during audio and video calls

Patent metadata

Field	Value
Publication number	US-12028302-B2
Application number	US-202318215221-A
Country	US
Kind code	B2
Filing date	Jun 28, 2023
Priority date	Jul 30, 2017
Publication date	Jul 2, 2024
Grant date	Jul 2, 2024

Abstract

Implementations relate to providing information items for display during a communication session. In some implementations, a computer-implemented method includes receiving, during a communication session between a first computing device and a second computing device, first media content from the communication session. The method further includes determining a first information item for display in the communication session based at least in part on the first media content. The method further includes sending a first command to at least one of the first computing device and the second computing device to display the first information item.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
receiving session content during a video communication session between a first computing device and a second computing device;
determining, based on the session content, that the session content includes a request for media from the first computing device;
outputting, with a trained machine-learning model, context from the session content;
altering the media based on the context; and
sending a first command to at least one of the first computing device or the second computing device to display the altered media.

2. The method of claim 1 , wherein:
the context includes an identification of emotions in the session content; and
altering the media based on the context includes altering the media based on the emotions in the session content.

3. The method of claim 1 , further comprising:
outputting, with the trained machine-learning model, a speech-to-text conversion of an audio portion of the session content.

4. The method of claim 1 , wherein the trained machine-learning model is personalized for a particular user by:
modifying one or more parameters for the trained machine-learning model based on the particular user;
saving the modified parameters for the trained machine-learning model; and
initializing the trained machine-learning model using the modified parameters for the particular user.

5. The method of claim 1 , wherein the trained machine-learning model is trained using training data and audio and video sources and wherein audio or video conversations that are not from the audio and video sources are excluded from the training data.

6. The method of claim 1 , wherein the trained machine-learning model outputs a determination that the session content includes the request for media based on a user associated with the first computing device making an explicit invocation for assistance in the session content.

7. The method of claim 6 , wherein the trained machine-learning model outputs the determination that the session content includes the request for media based on at least one selected from the group of a gender of a user associated with the first computing device, an estimated age of the user, an identification of the user being indoors or outdoors, a language spoken by the user, and combinations thereof.

8. The method of claim 1 , further comprising:
outputting, with the trained machine-learning model, at least one determination selected from the group of whether a face is present in the session content, a position of the face in the session content, a number of faces in the session content, and combinations thereof.

9. The method of claim 1 , further comprising:
analyzing the session content using a video analysis technique to determine that a user associated with the first computing device explicitly requests the media;
wherein the video analysis technique is at least one selected from the group of face detection, motion detection, gesture detection, and combinations thereof.

10. The method of claim 1 , further comprising:
generating, with the trained machine-learning model, the media in response to the request for the media from the first computing device.

11. A computing device comprising:
a processor; and
a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
receiving session content during a video communication session between a first computing device and a second computing device;
determining, based on the session content, that the session content includes a request for media from the first computing device;
outputting, with a trained machine-learning model, context from the session content;
altering the media based on the context; and
sending a first command to at least one of the first computing device or the second computing device to display the altered media.

12. The computing device of claim 11 , wherein:
the context includes an identification of emotions in the session content; and
altering the media based on the context includes altering the media based on the emotions in the session content.

13. The computing device of claim 11 , wherein the operations further comprise:
outputting, with the trained machine-learning model, a speech-to-text conversion of an audio portion of the session content.

14. The computing device of claim 11 , wherein the trained machine-learning model is personalized for a particular user by:
modifying one or more parameters for the trained machine-learning model based on the particular user;
saving the modified parameters for the trained machine-learning model; and
initializing the trained machine-learning model using the modified parameters for the particular user.

15. The computing device of claim 11 , wherein the trained machine-learning model is trained using training data and audio and video sources and wherein audio or video conversations that are not from the audio and video sources are excluded from the training data.

16. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
receiving session content during a video communication session between a first computing device and a second computing device;
determining, based on the session content, that the session content includes a request for media from the first computing device;
outputting, with a trained machine-learning model, context from the session content;
altering the media based on the context; and
sending a first command to at least one of the first computing device or the second computing device to display the altered media.

17. The non-transitory computer-readable medium of claim 16 , wherein:
the context includes an identification of emotions in the session content; and
altering the media based on the context includes altering the media based on the emotions in the session content.

18. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
outputting, with the trained machine-learning model, a speech-to-text conversion of an audio portion of the session content.

19. The non-transitory computer-readable medium of claim 16 , wherein the trained machine-learning model is personalized for a particular user by:
modifying one or more parameters for the trained machine-learning model based on the particular user;
saving the modified parameters for the trained machine-learning model; and
initializing the trained machine-learning model using the modified parameters for the particular user.

20. The non-transitory computer-readable medium of claim 16 , wherein the trained machine-learning model is trained using training data and audio and video sources and wherein audio or video conversations that are not from the audio and video sources are excluded from the training data.

Assignees

• Google Llc

Inventors

• Bergenlid Fredrik
• Lysychkin Vladyslav
• Burakov Denis
• Behzadi Behshad
• Van Scheltinga Andrea Terwisscha
• De Laroussilhe Quentin Lascombes
• Golikov Mikhail
• Metter Koa
• Badr Ibrahim
• Sabur Zaheed

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• H04N21/4394

  involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58) · CPC title

• G10L15/005

  Language recognition · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• H04N21/4788

  communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)} · CPC title