Assistance during audio and video calls

Patent metadata

Field	Value
Publication number	US-11765113-B2
Application number	US-202217991300-A
Country	US
Kind code	B2
Filing date	Nov 21, 2022
Priority date	Jul 30, 2017
Publication date	Sep 19, 2023
Grant date	Sep 19, 2023

Abstract

Implementations relate to providing information items for display during a communication session. In some implementations, a computer-implemented method includes receiving, during a communication session between a first computing device and a second computing device, first media content from the communication session. The method further includes determining a first information item for display in the communication session based at least in part on the first media content. The method further includes sending a first command to at least one of the first computing device and the second computing device to display the first information item.

First claim

The invention claimed is:

1 . A computer-implemented method comprising:
determining a user preference associated with a user for a particular virtual assistant from a set of assistants; receiving, during a video communication session between a first computing device associated with the user and a second computing device, first session content from the video communication session; detecting that the first session content includes a request for media; selecting the particular virtual assistant based on the user preference; and sending, by the particular virtual assistant, a first command to at least one of the first computing device or the second computing device to display the media.

2 . The method of claim 1 , further comprising:
requesting, by the particular virtual assistant, output from a second virtual assistant of the set of assistants, wherein the second virtual assistant provides a different service than the particular virtual assistant is operable to provide.

3 . The method of claim 2 , wherein the different service is translation of text-to-speech to provide speech output in a target language that is understood by the user.

4 . The method of claim 1 , wherein determining the user preference associated with the user is based on the user explicitly providing the user preference for the particular virtual assistant.

5 . The method of claim 1 , wherein determining the user preference associated with the user is based on at least one selected from the group of user feedback, the user performing an action based on the particular virtual assistant sending the first command, the user choosing an option from the particular virtual assistant that is not offered by other virtual assistants in the set of assistants, the user providing an indication of user satisfaction, and combinations thereof.

6 . The method of claim 1 , wherein the video communication session includes video that includes a face, and wherein the first command causes display of the media such that the face is not obscured.

7 . The method of claim 1 , wherein detecting that the first session content includes the request for media is based on determining from conversation context of the video communication session that an implicit invocation of the particular virtual assistant is associated with a confidence score that exceeds a score threshold.

8 . A computing device comprising:
a processor; and a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
determining a user preference associated with a user for a particular virtual assistant from a set of assistants;
receiving, during a video communication session between a first computing device associated with the user and a second computing device, first session content from the video communication session;
detecting that the first session content includes a request for media;
selecting the particular virtual assistant based on the user preference; and
sending, by the particular virtual assistant, a first command to at least one of the first computing device or the second computing device to display the media.

9 . The computing device of claim 8 , wherein the operations further comprise:
requesting, by the particular virtual assistant, output from a second virtual assistant of the set of assistants, wherein the second virtual assistant provides a different service than the particular virtual assistant is operable to provide.

10 . The computing device of claim 9 , wherein the different service is translation of text-to-speech to provide speech output in a target language that is understood by the user.

11 . The computing device of claim 8 , wherein determining the user preference associated with the user is based on the user explicitly providing the user preference for the particular virtual assistant.

12 . The computing device of claim 8 , wherein determining the user preference associated with the user is based on at least one selected from the group of user feedback, the user performing an action based on the particular virtual assistant sending the first command, the user choosing an option from the particular virtual assistant that is not offered by other virtual assistants in the set of assistants, the user providing an indication of user satisfaction, and combinations thereof.

13 . The computing device of claim 8 , wherein the video communication session includes video that includes a face, and wherein the first command causes display of the media such that the face is not obscured.

14 . The computing device of claim 8 , wherein detecting that the first session content includes the request for media is based on determining from conversation context of the video communication session that an implicit invocation of the particular virtual assistant is associated with a confidence score that exceeds a score threshold.

15 . A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
determining a user preference associated with a user for a particular virtual assistant from a set of assistants; receiving, during a video communication session between a first computing device associated with the user and a second computing device, first session content from the video communication session; detecting that the first session content includes a request for media; selecting the particular virtual assistant based on the user preference; and sending, by the particular virtual assistant, a first command to at least one of the first computing device or the second computing device to display the media.

16 . The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise:
requesting, by the particular virtual assistant, output from a second virtual assistant of the set of assistants, wherein the second virtual assistant provides a different service than the particular virtual assistant is operable to provide.

17 . The non-transitory computer-readable medium of claim 16 , wherein the different service is translation of text-to-speech to provide speech output in a target language that is understood by the user.

18 . The non-transitory computer-readable medium of claim 15 , wherein determining the user preference associated with the user is based on the user explicitly providing the user preference for the particular virtual assistant.

19 . The non-transitory computer-readable medium of claim 15 , wherein determining the user preference associated with the user is based on at least one selected from the group of user feedback, the user performing an action based on the particular virtual assistant sending the first command, the user choosing an option from the particular virtual assistant that is not offered by other virtual assistants in the set of assistants, the user providing an indication of user satisfaction, and combinations thereof.

20 . The non-transitory computer-readable medium of claim 15 , wherein the video communication session includes video that includes a face, and wherein the first command causes display of the media such that the face is not obscured.

Assignees

• Google Llc

Inventors

• Bergenlid Fredrik
• Lysychkin Vladyslav
• Burakov Denis
• Behzadi Behshad
• Van Scheltinga Andrea Terwisscha
• De Laroussilhe Quentin Lascombes
• Golikov Mikhail
• Metter Koa
• Badr Ibrahim
• Sabur Zaheed

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• H04L51/10 Primary

  Multimedia information · CPC title

• G10L15/005

  Language recognition · CPC title

• G10L25/63

  for estimating an emotional state · CPC title

• G10L15/16

  using artificial neural networks · CPC title