Orchestrating execution of a series of actions requested to be performed via an automated assistant

Patent metadata

Field	Value
Publication number	US-11031007-B2
Application number	US-201916343285-A
Country	US
Kind code	B2
Filing date	Feb 7, 2019
Priority date	Nov 21, 2018
Publication date	Jun 8, 2021
Grant date	Jun 8, 2021

Abstract

Implementations are set forth herein for creating an order of execution for actions that were requested by a user, via a spoken utterance to an automated assistant. The order of execution for the requested actions can be based on how each requested action can, or is predicted to, affect other requested actions. In some implementations, an order of execution for a series of actions can be determined based on an output of a machine learning model, such as a model that has been trained according to supervised learning. A particular order of execution can be selected to mitigate waste of processing, memory, and network resources—at least relative to other possible orders of execution. Using interaction data that characterizes past performances of automated assistants, certain orders of execution can be adapted over time, thereby allowing the automated assistant to learn from past interactions with one or more users.

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
determining that a user has provided a spoken utterance that includes requests for an automated assistant to perform multiple actions that include a first type of action and a second type of action, wherein the automated assistant is accessible to the user via an automated assistant interface of a computing device;
generating, in response to the user providing the spoken utterance,
an estimated delay for the first type of action when the second type of action is prioritized over the first type of action during execution of the multiple actions;

determining, based on the estimated delay, whether the estimated delay for the first type of action satisfies a threshold,
wherein, when the estimated delay for the first type of action satisfies the threshold, execution of the first type of action is prioritized over the second type of action;

generating, based on whether the estimated delay satisfies the threshold, a preferred order of execution for the multiple actions requested by the user; and
causing the automated assistant to initialize performance of the multiple actions according to the preferred order of execution.

2. The method of claim 1 , further comprising:
determining an action classification for each action of the multiple actions requested by the user, wherein the automated assistant is configured to prioritize at least one particular classification of actions over at least one other classification of actions.

3. The method of claim 1 , wherein the first type of action includes a dialog initiating action and the second type of action includes a media playback action.

4. The method of claim 3 , wherein the media playback action is configured to be at least partially performed at a separate computing device, and the method further comprises:
when the dialog initiating action is prioritized over the media playback action:
causing the dialog initiating action to be initialized at the computing device simultaneous to causing the separate device to initialize an application for executing the media playback action.

5. The method of claim 4 , further comprising:
when the media playback action is prioritized over the dialog initiating action:
causing the automated assistant to provide a natural language output corresponding to dialog in furtherance of completing the dialog initiating action, and
when the dialog initiating action is completed:
causing the automated assistant to initialize performance of the media playback action at the computing device or the separate computing device.

6. The method of claim 3 , wherein the dialog initiating action, when executed, includes initializing a dialog session between the user and the automated assistant in order for the user to identify a value to be assigned to a parameter in furtherance of completing the dialog initiating action.

7. The method of claim 3 , wherein the media playback action, when executed, includes initializing playback of media that is accessible via one or more files, and the estimated delay is based on a total of file lengths for the one or more files.

8. A system, comprising:
one or more processors; and
memory configured to store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations that include:
determining that a user has provided a spoken utterance that includes requests for an automated assistant to perform multiple actions that include a first type of action and a second type of action, wherein the automated assistant is accessible to the user via an automated assistant interface of a computing device;
generating, in response to the user providing the spoken utterance, an estimated delay for the first type of action when the second type of action is prioritized over the first type of action during execution of the multiple actions;
determining, based on the estimated delay, whether the estimated delay for the first type of action satisfies a threshold, wherein, when the estimated delay for the first type of action satisfies the threshold, execution of the first type of action is prioritized over the second type of action;
generating, based on whether the estimated delay satisfies the threshold, a preferred order of execution for the multiple actions requested by the user; and
causing the automated assistant to initialize performance of the multiple actions according to the preferred order of execution.

9. The system of claim 8 , wherein the operations further include:
determining an action classification for each action of the multiple actions requested by the user, wherein the automated assistant is configured to prioritize at least one particular classification of actions over at least one other classification of actions.

10. The system of claim 8 , wherein the first type of action includes a dialog initiating action and the second type of action includes a media playback action.

11. The system of claim 10 , wherein the media playback action is configured to be at least partially performed at a separate computing device, and wherein the operations further include:
when the dialog initiating action is prioritized over the media playback action:
causing the dialog initiating action to be initialized at the computing device simultaneous to causing the separate device to initialize an application for executing the media playback action.

12. The system of claim 11 , wherein the operations further include:
when the media playback action is prioritized over the dialog initiating action:
causing the automated assistant to provide a natural language output corresponding to dialog in furtherance of completing the dialog initiating action, and
when the dialog initiating action is completed:
causing the automated assistant to initialize performance of the media playback action at the computing device or the separate computing device.

13. The system of claim 10 , wherein the dialog initiating action, when executed, includes initializing a dialog session between the user and the automated assistant in order for the user to identify a value to be assigned to a parameter in furtherance of completing the dialog initiating action.

14. The system of claim 10 , wherein the media playback action, when executed, includes initializing playback of media that is accessible via one or more files, and the estimated delay is based on a total of file lengths for the one or more files.

Assignees

• Google Llc

Inventors

• Andreica Mugurel Ionut
• Vuskovic Vladimir
• Lange Joseph
• Stovezky Sharon
• Nowak-Przygodzki Marcin

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06Q10/10

  Office automation; Time management · CPC title

• G06Q30/015

  Providing customer assistance, e.g. assisting a customer within a business location or via helpdesk · CPC title

• G06Q50/10

  Services · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title