Multimodal transmission of packetized data

Patent metadata

Field	Value
Publication number	US-11381609-B2
Application number	US-202016909375-A
Country	US
Kind code	B2
Filing date	Jun 23, 2020
Priority date	Dec 30, 2016
Publication date	Jul 5, 2022
Grant date	Jul 5, 2022

Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

First claim

What is claimed is:

1. A system for multi-modal data transmission in a voice-based computing environment, comprising:
a data processing system comprising hardware circuitry and memory to:
receive, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a computing device;
parse the input audio signal to identify a request and a trigger keyword corresponding to the request;
generate, based on at least one of the request and the trigger keyword, a first action data structure;
select, based on at least one of the request and the trigger keyword, a content item via a real-time content selection process;
identify a first candidate interface and a second candidate interface within a threshold distance of the computing device;
select, based on a policy, the first candidate interface as a selected interface to present the content item;
provide the content item for delivery in a modality compatible with the selected interface selected based on the policy; and
transmit the content item in the modality to deliver the content item for presentation via the selected interface.

2. The system of claim 1 , wherein the policy comprises a heuristic-based policy, comprising:
the data processing system to select, based on the heuristic-based policy, the first candidate interface as the selected interface.

3. The system of claim 1 , wherein the policy comprises a resource utilization policy, comprising:
the data processing system to select, based on the resource utilization policy, the first candidate interface as the selected interface.

4. The system of claim 1 , comprising the data processing system to:
determine a first resource utilization value for the first candidate interface and a second resource utilization value for the second candidate interface; and
select, based on a comparison of the first resource utilization value with the second resource utilization value, the first candidate interface as the selected interface.

5. The system of claim 1 , comprising:
the data processing system to poll a plurality of interfaces to identify the first candidate interface and the second candidate interface within the threshold distance of the computing device.

6. The system of claim 1 , comprising:
the data processing system to instruct the computing device to transmit a short-range wireless transmission comprising a signal to identify a second computing device within the threshold distance of the computing device, wherein the second computing device comprises the second candidate interface.

7. The system of claim 1 , comprising:
the data processing system to identify, based on a plurality of proximity detection techniques, the first candidate interface and the second candidate interface.

8. The system of claim 1 , comprising the data processing system to:
identify, based on a plurality of proximity detection techniques, a plurality of computing devices within the threshold distance of the computing device, the plurality of computing devices comprising a second computing device comprising the second candidate interface and a third computing device comprising a third candidate interface;
determine a first resource utilization value for the first candidate interface, a second resource utilization value for the second candidate interface, and a third resource utilization value for the third candidate interface; and
select, based on a comparison of the first resource utilization value, the second resource utilization value, and the third resource utilization value, the first candidate interface as the selected interface to present the content item.

9. The system of claim 1 , comprising the data processing system to:
retrieve, from memory, proximity information identifying a plurality of computing devices within the threshold distance of the computing device; and
identify the first candidate interface and the second candidate interface as corresponding to one or more of the plurality of computing devices.

10. The system of claim 1 , comprising:
the data processing system to instruct the computing device to transmit a sub-audible wave to identify a second computing device within the threshold distance of the computing device, wherein the second computing device comprises the second candidate interface.

11. The system of claim 1 , wherein the computing device is a first client computing device, comprising:
the data processing system to determine that the first candidate interface corresponds to the first client computing device linked to an account, and that the second candidate interface corresponds to a second computing device linked to the account.

12. The system of claim 1 , wherein the first candidate interface includes an audio interface, and the second candidate interface includes a display screen.

13. A method for multi-modal data transmission in a voice-based computing environment, comprising:
receiving, by a data processing system comprising hardware circuitry and memory, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a computing device;
parsing, by the data processing system, the input audio signal to identify a request and a trigger keyword corresponding to the request;
generating, by the data processing system based on at least one of the request and the trigger keyword, a first action data structure;
selecting, by the data processing system, based on at least one of the request and the trigger keyword, a content item via a real-time content selection process;
identifying, by the data processing system, a first candidate interface and a second candidate interface within a threshold distance of the computing device;
selecting, by the data processing system based on a policy, the first candidate interface as a selected interface to present the content item;
providing, by the data processing system, the content item for delivery in a modality compatible with the selected interface selected based on the policy; and
transmitting, by the data processing system, the content item in the modality to deliver the content item for presentation via the selected interface.

14. The method of claim 13 , wherein the policy comprises a heuristic-based policy, comprising:
selecting, by the data processing system based on the heuristic-based policy, the first candidate interface as the selected interface.

15. The method of claim 13 , wherein the policy comprises a resource utilization policy, comprising:
selecting, by the data processing system based on the resource utilization policy, the first candidate interface as the selected interface.

16. The method of claim 13 , comprising:
determining, by the data processing system, a first resource utilization value for the first candidate interface and a second resource utilization value for the second candidate interface; and
selecting, by the data processing system based on a comparison of the first resource utilization value with the second resource utilization value, the first candidate interface as the selected interface.

17. The method of claim 13 , comprising:
identifying, by the data processing system based on a plurality of proximity detection techniques, the first candidate interface and the second candidate interface.

18. The method of claim 13 , comprising:
identifying, by the data processing system based on a plurality of proximity detection techniques, a plurality of computing devices within the threshold distance of the computing device, the plura

Assignees

• Google Llc

Inventors

• Lewis Justin
• Rapp Richard
• Bhaya Gaurav
• Stets Robert

Classifications

• G06F16/3343

  using phonetics · CPC title

• G06F16/3329

  Natural language query formulation · CPC title

• G06F9/5027 Primary

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G01S2205/02

  Indoor · CPC title

• G01S11/14

  using ultrasonic, sonic or infrasonic waves · CPC title