Orchestrating execution of a series of actions requested to be performed via an automated assistant

Patent metadata

Field	Value
Publication number	US-11769502-B2
Application number	US-202117339114-A
Country	US
Kind code	B2
Filing date	Jun 4, 2021
Priority date	Nov 21, 2018
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

Implementations are set forth herein for creating an order of execution for actions that were requested by a user, via a spoken utterance to an automated assistant. The order of execution for the requested actions can be based on how each requested action can, or is predicted to, affect other requested actions. In some implementations, an order of execution for a series of actions can be determined based on an output of a machine learning model, such as a model that has been trained according to supervised learning. A particular order of execution can be selected to mitigate waste of processing, memory, and network resources—at least relative to other possible orders of execution. Using interaction data that characterizes past performances of automated assistants, certain orders of execution can be adapted over time, thereby allowing the automated assistant to learn from past interactions with one or more users.

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
receiving audio data that that characterizes a spoken utterance from a user, wherein the spoken utterance includes a request for multiple actions to be performed via an automated assistant and the spoken utterance is received at an automated assistant interface of a computing device;
identifying, based on the audio data characterizing the spoken utterance, each action of the multiple actions requested by the user to be performed via the automated assistant, wherein requests for the multiple actions to be performed are set forth in the spoken utterance according to a first order of actions;
determining, based on identifying each action of the multiple actions, an execution characteristic of each action of the multiple actions,
wherein a particular execution characteristic of an action of the multiple actions affects a temporal aspect of execution of the multiple actions when the multiple actions are executed according to the first order of actions by one or more computing devices, and
wherein determining the execution characteristic of each action of the multiple actions includes accessing data that is generated based on past executions of one or more actions of the multiple actions at the computing device and/or a separate computing device;

determining, based on the particular execution characteristic of the action of the multiple actions, a second order of actions for executing the multiple actions, wherein the second order of actions, when executed by the one or more computing devices, causes the one or more computing devices to exhibit a different temporal aspect of execution of the multiple actions; and
causing, based on determining the second order of actions, the automated assistant to initialize performance of one or more actions of the multiple actions according to the second order of actions.

2. The method of claim 1 , wherein determining the second order of actions includes:
processing output data from a trained neural network model, the trained neural network model having been trained using historical interaction data that characterizes at least one or more previous interactions between the user and the automated assistant.

3. The method of claim 2 , wherein the historical interaction data further characterizes multiple interactions involving other users that have previously interacted with the automated assistant in furtherance of causing the automated assistant to perform various sequences of actions.

4. The method of claim 2 , wherein the historical interaction data further characterizes feedback provided by the user to the automated assistant in order to influence an order of execution of previously requested actions.

5. The method of claim 1 , wherein the particular execution characteristic of the action of the multiple actions characterizes the action as a dialog initiating action, and wherein a supplemental dialog session between the user and the automated assistant is to occur for the user to identify a value to be assigned to a parameter of the action.

6. The method of claim 5 , wherein the temporal aspect of the execution of the multiple actions, according to the first order of actions, includes at least an estimated time of execution for one or more actions of the multiple actions, and wherein the method further comprises:
determining that the supplemental dialog session is predicted to extend the estimated time of execution for the one or more actions when the multiple actions are executed according to the first order of actions.

7. The method of claim 5 ,
wherein another action of the multiple actions includes providing continuous media playback, and
wherein the second order of the actions prioritizes the dialog initiating action over the other action that includes providing the continuous media playback.

8. The method of claim 5 , wherein causing the automated assistant to initialize performance of the at least one action of the multiple actions according to the second order of actions includes:
generating a natural language output that provides the user with an indication that the at least one action of the multiple actions has been initialized according to the second order of actions.

9. A system comprising:
memory storing instructions;
one or more processors operable to execute the instructions to:
receive audio data that that characterizes a spoken utterance from a user, wherein the spoken utterance includes a request for multiple actions to be performed via an automated assistant and the spoken utterance is received at an automated assistant interface of a computing device;
identify, based on the audio data characterizing the spoken utterance, each action of the multiple actions requested by the user to be performed via the automated assistant, wherein requests for the multiple actions to be performed are set forth in the spoken utterance according to a first order of actions;
determine, based on identifying each action of the multiple actions, an execution characteristic of each action of the multiple actions,
wherein a particular execution characteristic of an action of the multiple actions affects a temporal aspect of execution of the multiple actions when the multiple actions are executed according to the first order of actions by one or more computing devices, and
wherein determining the execution characteristic of each action of the multiple actions includes accessing data that is generated based on past executions of one or more actions of the multiple actions at the computing device and/or a separate computing device;

determine, based on the particular execution characteristic of the action of the multiple actions, a second order of actions for executing the multiple actions, wherein the second order of actions, when executed by the one or more computing devices, causes the one or more computing devices to exhibit a different temporal aspect of execution of the multiple actions; and
cause, based on determining the second order of actions, the automated assistant to initialize performance of one or more actions of the multiple actions according to the second order of actions.

10. The system of claim 9 , wherein the instructions to determine the second order of actions include:
process output data from a trained neural network model, the trained neural network model having been trained using historical interaction data that characterizes at least one or more previous interactions between the user and the automated assistant.

11. The system of claim 10 , wherein the historical interaction data further characterizes multiple interactions involving other users that have previously interacted with the automated assistant in furtherance of causing the automated assistant to perform various sequences of actions.

12. The system of claim 10 , wherein the historical interaction data further characterizes feedback provided by the user to the automated assistant in order to influence an order of execution of previously requested actions.

13. The system of claim 9 , wherein the particular execution characteristic of the action of the multiple actions characterizes the action as a dialog initiating action, and wherein a supplemental dialog session between the user and the automated assistant is to occur for the user to identify a value to be assigned to a parameter of the action.

14. The system of claim 13 , wherein the temporal aspect of the execution of the multiple actions, according to the first order of actions, includes at least an estimated time of execution for one or more actions of the multiple actio

Assignees

• Google Llc

Inventors

• Andreica Mugurel Ionut
• Vuskovic Vladimir
• Lange Joseph
• Stovezky Sharon
• Nowak-Przygodzki Marcin

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06Q10/10

  Office automation; Time management · CPC title

• G06Q30/015

  Providing customer assistance, e.g. assisting a customer within a business location or via helpdesk · CPC title

• G06Q50/10

  Services · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title