Multimodal transmission of packetized data

Patent metadata

Field	Value
Publication number	US-11705121-B2
Application number	US-202016936972-A
Country	US
Kind code	B2
Filing date	Jul 23, 2020
Priority date	Dec 30, 2016
Publication date	Jul 18, 2023
Grant date	Jul 18, 2023

Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

First claim

What is claimed is:

1. A system, comprising:
a data processing system comprising memory and one or more processors to:
receive data packets comprising voice input detected by a sensor of a computing device;
parse the voice input to identify a keyword;
select, based on the keyword, one or more content items;
identify a first candidate interface and a second candidate interface associated with the computing device;
select, based on a characteristic associated with the first candidate interface and the second candidate interface, the first candidate interface as a selected interface to present a first content item of the one or more content items; and
transmit, to the computing device, the first content item for presentation via the selected interface in a modality compatible with the selected interface selected based on the characteristic.

2. The system of claim 1 , comprising the data processing system to:
generate, based on the keyword, one or more action data structures; and
transmit a first action data structure in the modality compatible with the selected interface.

3. The system of claim 1 , comprising the data processing system to:
identify a request in the voice input;
generate a first action data structure based on the request and the keyword; and
select the first candidate interface based on the first action data structure.

4. The system of claim 3 , comprising the data processing system to:
select the first candidate interface based on at least one of a category or an output type of the first action data structure.

5. The system of claim 1 , comprising the data processing system to:
parse the voice input to identify the keyword and a request;
generate a first action data structure based on the request and the keyword;
select the one or more content items based on the request and the keyword; and
provide the first action data structure and the first content item for delivery in the modality compatible with the selected interface selected based on the characteristic.

6. The system of claim 1 , comprising:
the data processing system to identify the first candidate interface and the second candidate interface located within a threshold distance of the computing device.

7. The system of claim 1 , wherein the first candidate interface, the second candidate interface, and the computing device are linked with an account, comprising:
the data processing system to identify the first candidate interface and the second candidate interface based on the first candidate interface and the second candidate interface being linked with the account.

8. The system of claim 1 , comprising the data processing system to:
poll a plurality of interfaces linked to an account of the computing device to identify a plurality of available interfaces, wherein the plurality of available interfaces comprises the first candidate interface and the second candidate interface.

9. The system of claim 1 , comprising the data processing system to:
identify a first characteristic associated with the first candidate interface, and a second characteristic associated with the second candidate interface; and
select the first candidate interface based on the first characteristic.

10. The system of claim 1 , comprising the data processing system to:
determine a first resource utilization value for the first candidate interface and a second resource utilization value for the second candidate interface; and
select, based on a comparison between the first resource utilization value and the second resource utilization value, the first candidate interface as the selected interface.

11. The system of claim 1 , comprising:
the data processing system to select the one or more content items via a real-time content selection process.

12. A method, comprising:
receiving, by a data processing system comprising memory and one or more processors, data packets comprising voice input detected by a sensor of a computing device;
parsing, by the data processing system, the voice input to identify a keyword;
selecting, by the data processing system based on the keyword, one or more content items;
identifying, by the data processing system, a first candidate interface and a second candidate interface associated with the computing device;
selecting, by the data processing system, based on a characteristic associated with the first candidate interface and the second candidate interface, the first candidate interface as a selected interface to present a first content item of the one or more content items; and
transmitting, by the data processing system to the computing device, the first content item for presentation via the selected interface in a modality compatible with the selected interface selected based on the characteristic.

13. The method of claim 12 , comprising:
generating, by the data processing system based on the keyword, one or more action data structures; and
transmitting, by the data processing system, a first action data structure in the modality compatible with the selected interface.

14. The method of claim 12 , comprising:
identifying, by the data processing system, a request in the voice input;
generating, by the data processing system, a first action data structure based on the request and the keyword; and
selecting, by the data processing system, the first candidate interface based on the first action data structure.

15. The method of claim 14 , comprising:
selecting, by the data processing system, the first candidate interface based on at least one of a category or an output type of the first action data structure.

16. The method of claim 12 , comprising:
parsing, by the data processing system, the voice input to identify the keyword and a request;
generating, by the data processing system, a first action data structure based on the request and the keyword;
selecting, by the data processing system, the one or more content items based on the request and the keyword; and
providing, by the data processing system, the first action data structure and the first content item for delivery in the modality compatible with the selected interface selected based on the characteristic.

17. The method of claim 12 , comprising:
identifying, by the data processing system, the first candidate interface and the second candidate interface located within a threshold distance of the computing device.

18. The method of claim 12 , wherein the first candidate interface, the second candidate interface, and the computing device are linked with an account, comprising:
identifying, by the data processing system, the first candidate interface and the second candidate interface based on the first candidate interface and the second candidate interface being linked with the account.

19. The method of claim 12 , comprising:
polling, by the data processing system, a plurality of interfaces linked to an account of the computing device to identify a plurality of available interfaces, wherein the plurality of available interfaces comprises the first candidate interface and the second candidate interface.

20. The method of claim 12 , comprising:
determining, by the data processing system, a first resource utilization value for the first candidate interface and a second resource utilization value for the second candidate interface; and
selecting, by the data processing system, based on a comparison between the first resource utilization value and the second resource utilization value, the first candidate interface

Assignees

• Google Llc

Inventors

• Bhaya Gaurav
• Stets Robert
• Patil Umesh

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F40/279

  Recognition of textual entities · CPC title

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G10L2015/088

  Word spotting · CPC title