Activation of remote devices in a networked system

Patent metadata

Field	Value
Publication number	US-11024306-B2
Application number	US-201816131453-A
Country	US
Kind code	B2
Filing date	Sep 14, 2018
Priority date	May 7, 2018
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

The present disclosure is generally directed to the generation of voice-activated data flows in interconnected network. The voice-activated data flows can include input audio signals that include a request and are detected at a client device. The client device can transmit the input audio signal to a data processing system, where the input audio signal can be parsed and passed to the data processing system of a service provider to fulfill the request in the input audio signal. The present solution is configured to conserve network resources by reducing the number of network transmissions needed to fulfill a request.

First claim

The invention claimed is:

1. A system to generate threads in a networked computer environment:
a data processing system executing a natural language processor component and an action handler component to:
receive, by the natural language processor component, a first input audio signal detected by a sensor of a client computing device;
parse, by the natural language processor component, the first input audio signal to identify a first request;
select, by the action handler component, a service provider device to fulfill the first request and an action data structure comprising a form definition in a schema-based format that indicates parameters required for fulfillment of the first request;
generate, by the action handler component, a node graph based on the form definition that indicates the parameters required for the fulfillment of the first request, the node graph including parameter nodes that correspond to the parameters required for the fulfillment of the first request and that indicate slots in a dialog frame that is based on the form definition and that is a data structure that indicates the parameters required for the fulfillment of the first request, and the node graph further including a validation node that indicates that a value of a parameter node in the node graph that is an input to the validation node is to be sent to and validated by the service provider device;
identify, by the action handler component, an open slot of the dialog frame based on the node graph, the open slot being one of the slots in the dialog frame that has an empty value;
generate, by the action handler component, a first dialog candidate based on the open slot of the dialog frame;
transmit, by the action handler component, the first dialog candidate to the client computing device;
receive, by the natural language processor component, a second input audio signal detected by the sensor of the client computing device in response to the first dialog candidate rendered at the client computing device;
parse, by the natural language processor component, the second input audio signal to identify a parameter;
store, by the action handler component, the parameter into the open slot of the dialog frame; and
transmit, by the action handler component, the action data structure to the service provider device to fulfill the first request.

2. The system of claim 1 , comprising the action handler component to:
store the action data structure with the parameter in the open slot of the dialog frame as a form state;
transmit the form state to the service provider device for validation of the parameter;
receive an updated form state; and
update the action data structure based on the updated form state.

3. The system of claim 1 , comprising the action handler component to:
transmit a form state comprising the parameter in the open slot of the dialog frame to the service provider device for validation by the service provider device;
receive an updated form state that indicates the parameter in the open slot of the dialog frame is invalid; and
generate a second dialog candidate based on the open slot of the dialog frame.

4. The system of claim 1 , comprising the action handler component to:
store a form state comprising the parameter in the open slot based on transmitting the action data structure to the service provider device.

5. The system of claim 1 , comprising:
store, by the action handler component and based on the transmission of the action data structure to the service provider device, a form state based on the first request;
receive, by the natural language processor component, a third input audio signal detected by the sensor of the client computing device, the third input audio signal comprising a second request;
select, by the action handler component, the service provider device configured to fulfill the second request and a second action data structure comprising associated with the second request;
fill, by the action handler component, at least one slot of the second action data structure based on the form state based on the first request.

6. The system of claim 1 , comprising the action handler component to:
identify each of a plurality of open slots in the dialog frame; and
generate a respective dialog candidate for each of the plurality of open slots in the dialog frame.

7. The system of claim 1 , comprising a remote application launcher executed by the data processing system to:
generate a digital component comprising an indication of an application associated with the service provider device and installed at the client computing device, the digital component configured to launch the application associated with the service provider device; and
transmit the digital component to the client computing device.

8. The system of claim 1 , comprising:
identify, by the action handler component, a second open slot of the dialog frame;
generate, by the action handler component, a second dialog candidate based on the second open slot of the dialog frame;
transmit, by the action handler component, the second dialog candidate to the client computing device;
receive, by the natural language processor component, a third input audio signal detected by the sensor of the client computing device in response to rendering the second dialog candidate at the client computing device;
parse, by the natural language processor component, the third input audio signal to identify an entity;
expand, by the action handler component, the entity into a format associated with the second open slot of the dialog frame based on an expansion policy associated with the client computing device; and
store, by the action handler component, the expanded entity into the second open slot of the dialog frame.

9. The system of claim 1 , comprising the action handler component to:
convert the parameter into a format associated with the open slot of the dialog frame.

10. The system of claim 1 , wherein:
the node graph is a directed acyclic graph, and
the parameter nodes are linked together to indicate a hierarchal relationship between the slots in the dialog frame, and a value of a slot that is lower in the hierarchal relationship depends on a value of a slot that is higher in the hierarchal relationship.

11. A method to generate communication threads in a networked computer environment:
receiving, by a natural language processor component executed by a data processing system, a first input audio signal detected by a sensor of a client computing device;
parsing, by the natural language processor component, the first input audio signal to identify a first request;
selecting, by an action handler component executed by the data processing system, a service provider device to fulfill the first request and an action data structure comprising a form definition in a schema-based format indicating parameters required for fulfillment of the first request;
generating, by the action handler component, a node graph based on the form definition that indicates the parameters required for the fulfillment of the first request, the node graph including parameter nodes that correspond to the parameters required for the fulfillment of the first request and that indicate slots in a dialog frame that is based on the form definition and that is a data structure that indicates the parameters required for the fulfillment of the first request, and the node graph further including a validation node that indicates that a value of a parameter node in the node graph that is an input to the validation node is to be sent to and validated by the service provider device;
identifying, by the action handler component, an open slot of t

Assignees

• Google Llc

Inventors

• Bhaya Gaurav
• Kirazci Ulas
• Abrams Bradley
• Coimbra Adam
• Firman Ilya
• Radebaugh Carey

Classifications

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G10L15/30

  Distributed recognition, e.g. in client-server systems, for mobile phones or network applications · CPC title

• G06F40/216

  using statistical methods · CPC title

• G10L2015/225

  Feedback of the input speech · CPC title