Assistance during audio and video calls

Patent metadata

Field	Value
Publication number	US-12348469-B2
Application number	US-202418670389-A
Country	US
Kind code	B2
Filing date	May 21, 2024
Priority date	Jul 30, 2017
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

Implementations relate to providing information items for display during a communication session. In some implementations, a computer-implemented method includes receiving, during a communication session between a first computing device and a second computing device, first media content from the communication session. The method further includes determining a first information item for display in the communication session based at least in part on the first media content. The method further includes sending a first command to at least one of the first computing device and the second computing device to display the first information item.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
receiving session video content during a video communication session between a first computing device and a second computing device;
detecting, in the session video content, a gesture performed by a user associated with the first computing device;
determining, with a trained machine-learning model and based on the gesture, that the user invoked a request for assistance, wherein the request comprises a request for media and wherein the trained machine-learning model outputs a confidence score associated with the determination;
in response to the confidence score meeting a threshold, outputting, by the trained machine-learning model, the media; and
sending a first command to at least one of the first computing device or the second computing device to display the media.

2. The method of claim 1 , wherein the first command causes the at least one of the first computing device or the second computing device to display the media in a user interface as an illustration that is overlaid atop the session video content.

3. The method of claim 2 , further comprising:
determining, with the trained machine-learning model, an emotion associated with the user during the video communication session;
wherein the illustration is selected based on the gesture and the emotion associated with the user during the video communication session.

4. The method of claim 2 , wherein the illustration is selected from a group of a heart, a birthday-related graphic, a trophy, and combinations thereof.

5. The method of claim 1 , further comprising:
performing speech-to-text translation of audio associated with the user using the video communication session to obtain text;
wherein determining that the user associated with the first computing device invoked the request for media is further based on the text.

6. The method of claim 1 , wherein the trained machine-learning model is personalized to the user by:
modifying one or more parameters of a pre-trained machine-learning model based on the user to obtain the trained machine-learning model.

7. The method of claim 1 , wherein determining that the user associated with the first computing device invoked the request for media is further based on at least one detection selected from a group of face detection, motion detection, and combinations thereof.

8. A computing device comprising:
a processor; and
a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
receiving session video content during a video communication session between a first computing device and a second computing device;
detecting, in the session video content, a gesture performed by a user associated with the first computing device;
determining, with a trained machine-learning model and based on the gesture, that the user invoked a request for assistance, wherein the request comprises a request for media and wherein the trained machine-learning model outputs a confidence score associated with the determination;
in response to the confidence score meeting a threshold, outputting, by the trained machine-learning model, the media; and
sending a first command to at least one of the first computing device or the second computing device to display the media.

9. The computing device of claim 8 , wherein the first command causes the at least one of the first computing device or the second computing device to display the requested media in a user interface as an illustration that is overlaid atop the session video content.

10. The computing device of claim 9 , wherein the operations further include:
determining, with the trained machine-learning model, an emotion associated with the user during the video communication session;
wherein the illustration is selected based on the gesture and the emotion associated with the user during the video communication session.

11. The computing device of claim 9 , wherein the illustration is selected from a group of a heart, a birthday-related graphic, a trophy, and combinations thereof.

12. The computing device of claim 8 , wherein the operations further include:
performing speech-to-text translation of audio associated with the user using the video communication session to obtain text;
wherein determining that the user associated with the first computing device invoked the request for media is further based on the text.

13. The computing device of claim 8 , wherein the trained machine-learning model is personalized to the user by:
modifying one or more parameters of a pre-trained machine-learning model based on the user to obtain the trained machine-learning model.

14. The computing device of claim 8 , wherein determining that the user associated with the first computing device invoked the request for media is further based on at least one detection selected from a group of face detection, motion detection, and combinations thereof.

15. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
receiving session video content during a video communication session between a first computing device and a second computing device;
detecting, in the session video content, a gesture performed by a user associated with the first computing device;
determining, with a trained machine-learning model and based on the gesture, that the user invoked a request for assistance, wherein the request comprises a request for media and wherein the trained machine-learning model outputs a confidence score associated with the determination;
in response to the confidence score meeting a threshold, outputting, by the trained machine-learning model, the media; and
sending a first command to at least one of the first computing device or the second computing device to display the media.

16. The non-transitory computer-readable medium of claim 15 , wherein the first command causes the at least one of the first computing device or the second computing device to display the requested media in a user interface as an illustration that is overlaid atop the session video content.

17. The non-transitory computer-readable medium of claim 16 , wherein the operations further include:
determining, with the trained machine-learning model, an emotion associated with the user during the video communication session;
wherein the illustration is selected based on the gesture and the emotion associated with the user during the video communication session.

18. The non-transitory computer-readable medium of claim 16 , wherein the illustration is selected from a group of a heart, a birthday-related graphic, a trophy, and combinations thereof.

19. The non-transitory computer-readable medium of claim 15 , wherein the operations further include:
performing speech-to-text translation of audio associated with the user using the video communication session to obtain text;
wherein determining that the user associated with the first computing device invoked the request for media is further based on the text.

20. The non-transitory computer-readable medium of claim 15 , wherein the trained machine-learning model is personalized to the user by:
modifying one or more parameters of a pre-trained machine-learning model based on the user to obtain the trained machine-learning model.

Assignees

• Google Llc

Inventors

• Bergenlid Fredrik
• Lysychkin Vladyslav
• Burakov Denis
• Behzadi Behshad
• Van Scheltinga Andrea Terwisscha
• De Laroussilhe Quentin Lascombes
• Golikov Mikhail
• Metter Koa
• Badr Ibrahim
• Sabur Zaheed

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• H04N21/4394

  involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58) · CPC title

• G10L15/005

  Language recognition · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• H04N21/4788

  communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)} · CPC title