Realtime bandwidth-based communication for assistant systems

Patent metadata

Field	Value
Publication number	US-11249774-B2
Application number	US-202017013394-A
Country	US
Kind code	B2
Filing date	Sep 4, 2020
Priority date	Apr 20, 2018
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

In one embodiment, a method includes initiating a communication session with a second client system associated with a second user via a communication network, wherein the communication session is initiated in a first modality, receiving a ping to the first client system from the communication network to evaluate available bandwidth on the communication network, estimating, by the first client system, an amount of bandwidth available on the communication network for use by the first client system, determining, by the first client system, the amount of bandwidth available on the communication network for use by the first client system is insufficient for the first modality, and switching the communication session with the second client system to a second modality by the first client system, wherein the second modality uses less bandwidth than the first modality.

First claim

What is claimed is:

1. A method comprising, by a first client system associated with a first user:
initiating, via a communication network, a communication session with a second client system associated with a second user, wherein the communication session is initiated in a first modality;
receiving, from the communication network, a ping to the first client system to evaluate available bandwidth on the communication network;
estimating, by the first client system, an amount of bandwidth available on the communication network for use by the first client system;
determining, by the first client system, the amount of bandwidth available on the communication network for use by the first client system is insufficient for the first modality; and
switching, by the first client system, the communication session with the second client system to a second modality, wherein the second modality uses less bandwidth than the first modality.

2. The method of claim 1 , wherein estimating the amount of bandwidth available on the communication network for use by the first client system is based on antenna strength associated with the first client system.

3. The method of claim 1 , wherein the second client system converts the communication session to the first modality.

4. The method of claim 1 , wherein the first modality comprises video, audio, or text.

5. The method of claim 1 , wherein the communication session comprises a video clip, an audio clip, or a text string.

6. The method of claim 1 , wherein the first modality is video, and wherein the method further comprises:
determining the amount of bandwidth available on the communication network for use by the first client system is sufficient for the communication session to be in audio.

7. The method of claim 6 , wherein the second modality is audio.

8. The method of claim 1 , wherein the first modality is video, and wherein the method further comprises:
determining the amount of bandwidth available on the communication network for use by the first client system is insufficient for the communication session to be in audio.

9. The method of claim 8 , wherein the second modality is text.

10. The method of claim 1 , wherein the first modality is audio and the second modality is text, and wherein switching the communication session with the second client system to the second modality is based on automated speech recognition by an audio speech recognition (ASR) module.

11. The method of claim 10 , further comprising:
identifying one or more sounds associated with the audio; and
adding one or more descriptions of each of the one or more sounds to the text.

12. The method of claim 1 , wherein the first modality is video and the second modality is audio, and wherein switching the communication session with the second client system to the second modality comprises:
identifying one or more vocalizations associated with the video; and
generating the audio based on the one or more vocalizations.

13. The method of claim 12 , further comprising:
identifying one or more gestures associated with the video;
generating an audio representation of the one or more gestures; and
adding the audio representation to the generated audio.

14. The method of claim 1 , wherein the first modality is video and the second modality is text, and wherein switching the communication session with the second client system to the second modality comprises:
identifying one or more vocalizations associated with the video; and
generating the text based on the one or more vocalizations by an audio speech recognition (ASR) module.

15. The method of claim 14 , further comprising:
identifying one or more gestures associated with the video;
generating one or more descriptions for each of the one or more gestures; and
adding the generated descriptions to the text.

16. The method of claim 1 , wherein the communication session is associated with a network interface that communicates with the communication network, and wherein the amount of bandwidth available is estimated based on one or more attributes of the network interface.

17. The method of claim 1 , wherein estimating the amount of bandwidth available comprises:
receiving, at the first client system, a predetermined quantity of data via the communication network; and
measuring a rate at which the predetermined quantity of data is received.

18. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
initiate, via a communication network, a communication session with a second client system associated with a second user, wherein the communication session is initiated in a first modality;
receive, from the communication network, a ping to the first client system to evaluate available bandwidth on the communication network;
estimate, by the first client system, an amount of bandwidth available on the communication network for use by the first client system;
determine by the first client system, the amount of bandwidth available on the communication network for use by the first client system is insufficient for the first modality; and
switch, by the first client system, the communication session with the second client system to a second modality, wherein the second modality uses less bandwidth than the first modality.

19. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
initiate, via a communication network, a communication session with a second client system associated with a second user, wherein the communication session is initiated in a first modality;
receive, from the communication network, a ping to the first client system to evaluate available bandwidth on the communication network;
estimate, by the first client system, an amount of bandwidth available on the communication network for use by the first client system;
determine by the first client system, the amount of bandwidth available on the communication network for use by the first client system is insufficient for the first modality; and
switch, by the first client system, the communication session with the second client system to a second modality, wherein the second modality uses less bandwidth than the first modality.

Assignees

• Facebook Inc

Inventors

• Schissel Jason
• Fadeev Aleksey Sergeyevich

Classifications

• G06F16/3329

  Natural language query formulation · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title