Multimodal transmission of packetized data

Patent metadata

Field	Value
Publication number	US-11930050-B2
Application number	US-202217856636-A
Country	US
Kind code	B2
Filing date	Jul 1, 2022
Priority date	Dec 30, 2016
Publication date	Mar 12, 2024
Grant date	Mar 12, 2024

Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

First claim

What is claimed is:

1. A computing system, comprising:
one or more processors; and
one or more non-transitory, computer-readable media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
determining a request based on one or more input signals received by a first computing device;
obtaining a content item selected via a real-time content selection process based at least in part on the request;
identifying a first candidate interface and a second candidate interface within a threshold distance of the first computing device;
communicating, using sub-audible signals, with at least one of the first candidate interface and the second candidate interface;
selecting, based on a policy, the first candidate interface to render the content item; and
transmitting, to the first candidate interface for rendering, the content item in a modality compatible with the first candidate interface.

2. The computing system of claim 1 , wherein the sub-audible signals encode a token for identifying the at least one of the first candidate interface and the second candidate interface.

3. The computing system of claim 1 , wherein the sub-audible signals encode a token for determining that the first candidate interface and the second candidate interface are within a threshold distance.

4. The computing system of claim 1 , wherein a distance associated with the first candidate interface or the second candidate interface is determined based at least in part on a time interval between a first sub-audible signal and a second sub-audible signal.

5. The computing system of claim 1 , wherein one or more of the sub-audible signals are output by the first computing device and received by the at least one of the first candidate interface and the second candidate interface.

6. The computing system of claim 1 , wherein the first candidate interface and the second candidate interface are audio interfaces.

7. The computing system of claim 1 , wherein the first candidate interface and the second candidate interface are in different rooms of a building.

8. The computing system of claim 1 , wherein communicating with the at least one of the first candidate interface and the second candidate interface comprises interfacing with an application programming interface of the at least one of the first candidate interface and the second candidate interface.

9. One or more non-transitory, computer-readable media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
determining a request based on one or more input signals received by a first computing device;
obtaining a content item selected via a real-time content selection process based at least in part on the request;
identifying a first candidate interface and a second candidate interface within a threshold distance of the first computing device;
communicating, using sub-audible signals, with at least one of the first candidate interface and the second candidate interface;
selecting, based on a policy, the first candidate interface to render the content item; and
transmitting, to the first candidate interface for rendering, the content item in a modality compatible with the first candidate interface.

10. The one or more non-transitory, computer-readable media of claim 9 , wherein the sub-audible signals encode a token for identifying the at least one of the first candidate interface and the second candidate interface.

11. The one or more non-transitory, computer-readable media of claim 9 , wherein the sub-audible signals encode a token for determining that the first candidate interface and the second candidate interface are within a threshold distance.

12. The one or more non-transitory, computer-readable media of claim 9 , wherein a distance associated with the first candidate interface or the second candidate interface is determined based at least in part on a time interval between a first sub-audible signal and a second sub-audible signal.

13. The one or more non-transitory, computer-readable media of claim 9 , wherein one or more of the sub-audible signals are output by the first computing device and received by the at least one of the first candidate interface and the second candidate interface.

14. The one or more non-transitory, computer-readable media of claim 9 , wherein the first candidate interface and the second candidate interface are audio interfaces.

15. The one or more non-transitory, computer-readable media of claim 9 , wherein the first candidate interface and the second candidate interface are in different rooms of a building.

16. The one or more non-transitory, computer-readable media of claim 9 , wherein communicating with the at least one of the first candidate interface and the second candidate interface comprises interfacing with an application programming interface of the at least one of the first candidate interface and the second candidate interface.

17. A method, comprising:
determining, by a computing system, a request based on one or more input signals received by a first computing device;
obtaining, by the computing system, a content item selected via a real-time content selection process based at least in part on the request;
identifying, by the computing system, a first candidate interface and a second candidate interface within a threshold distance of the first computing device;
communicating, by the computing system and using sub-audible signals, with at least one of the first candidate interface and the second candidate interface;
selecting, by the computing system and based on a policy, the first candidate interface to render the content item; and
transmitting, by the computing system and to the first candidate interface for rendering, the content item in a modality compatible with the first candidate interface.

18. The method of claim 17 , wherein the sub-audible signals encode a token for identifying the at least one of the first candidate interface and the second candidate interface.

19. The method of claim 17 , wherein the sub-audible signals encode a token for determining that the first candidate interface and the second candidate interface are within a threshold distance.

20. The method of claim 17 , wherein a distance associated with the first candidate interface or the second candidate interface is determined based at least in part on a time interval between a first sub-audible signal and a second sub-audible signal.

Assignees

• Google Llc

Inventors

• Lewis Justin
• Rapp Richard
• Bhaya Gaurav
• Stets Robert

Classifications

• H04L65/1066 Primary

  Session management · CPC title

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G01S5/0295

  Proximity-based methods, e.g. position inferred from reception of particular signals · CPC title

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06F9/505

  considering the load · CPC title