Performing subtask(s) for a predicted action in response to a separate user interaction with an automated assistant prior to performance of the predicted action

Patent metadata

Field	Value
Publication number	US-11222637-B2
Application number	US-201916613705-A
Country	US
Kind code	B2
Filing date	May 31, 2019
Priority date	May 6, 2019
Publication date	Jan 11, 2022
Grant date	Jan 11, 2022

Abstract

Implementations herein relate to pre-caching data, corresponding to predicted interactions between a user and an automated assistant, using data characterizing previous interactions between the user and the automated assistant. An interaction can be predicted based on details of a current interaction between the user and an automated assistant. One or more predicted interactions can be initialized, and/or any corresponding data pre-cached, prior to the user commanding the automated assistant in furtherance of the predicted interaction. Interaction predictions can be generated using a user-parameterized machine learning model, which can be used when processing input(s) that characterize a recent user interaction with the automated assistant. Should the user command the automated assistant in a way that is aligned with a pre-cached, predicted interaction, the automated assistant will exhibit instant fulfillment of the command, thereby eliminating any latency that the user would have otherwise experienced interacting with the automated assistant.

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
determining, at a computing device that provides access to an automated assistant, that a user has provided a spoken utterance directed at the automated assistant;
generating, based on determining that the user has provided the spoken utterance, one or more action predictions corresponding to one or more actions that are predicted to be initialized,
wherein generating the one or more action predictions includes determining, for each respective action of the one or more actions, a probability that performance of the respective action of the one or more actions will be requested;

accessing, based on the one or more action predictions and prior to a user providing a subsequent input associated with the one or more actions, action advancement data corresponding to the one or more actions,
wherein the action advancement data corresponds to a type of data that is employed by the computing device and/or another computing device when performing a subtask of a given action of the one or more actions, and wherein the given action includes the subtask and other remaining subtasks;

performing, using the action advancement data, the subtask of the given action of the one or more actions, wherein the subtask is in addition to any caching of the action advancement data;
determining, subsequent to or while performing the subtask, whether the user provided another input in furtherance of the given action of the one or more actions; and
when the user is determined to have provided the other input in furtherance of the given action:
causing, in response to determining the other input was provided by the user, the other remaining subtasks of the given action of the one or more actions to be performed such that performance of the subtask of the given action is bypassed based on the subtask being previously performed.

2. The method of claim 1 , wherein the action advancement data includes device authentication data that is used when performing the given action to establish a connection between the computing device and a server device over a wide area network.

3. The method of claim 1 , wherein the action advancement data includes device connection data that is used when performing the given action to establish a connection between the computing device and a separate client device over a local area network.

4. The method of claim 1 , further comprising:
when the user is determined to have not provided the other input within a threshold period of time:
causing a trained machine learning model to be modified in response to the user not providing the other input within the threshold period of time, and
bypassing retrieving additional action advancement data in furtherance of completing one or more subtasks of the one or more actions.

5. The method of claim 4 , wherein bypassing retrieving additional action advancement data comprises actively dropping an HTTP connection between the computing device and a server device.

6. The method of claim 1 , wherein generating one or more action predictions corresponding to the one or more actions that the user is predicted to initialize performance of includes:
determining, for each respective action of the one or more actions, an estimated computational obligation for the computing device to complete performance of the respective action.

7. The method of claim 6 , wherein the estimated computational obligation comprises an estimated time period for the computing device to complete performance of the respective action, and wherein a longer estimated time period is associated with a higher estimated computational load for the respective action.

8. The method of claim 6 , further comprising:
assigning, based on a particular estimated computational obligation determined for the given action, an expiration time for the user and/or another user to request performance of the given action, wherein the expiration time corresponds to a particular time after the user provided the spoken utterance.

9. The method of claim 8 , wherein the length of the assigned expiration time is proportional to the estimated computational obligation determined for the given action.

10. The method of claim 1 , wherein generating the one or more action predictions corresponding to the one or more actions that the user is predicted to initialize performance of includes:
determining, for each action prediction of the one or more action predictions, whether a total number of action predictions satisfies an action prediction threshold, and
when the total number of action predictions satisfies the action prediction threshold:
bypassing generating additional action predictions based on the total number of action predictions satisfying the prediction threshold.

11. A method implemented by one or more processors, the method comprising:
determining, at a computing device that provides access to an automated assistant, that a user has provided one or more inputs directed to the automated assistant to initialize performance of an available action;
generating, in response to determining that the user has provided the one or more inputs, one or more action predictions corresponding to one or more actions that are predicted to be initialized,
wherein the one or more action predictions are generated by at least processing, using a trained machine learning model, an input that identifies at least the available action that the user requested the automated assistant to initialize performance of;

accessing, based on the one or more action predictions and prior to the user providing a subsequent input corresponding to a request for performance of a given action of the one or more actions, action advancement data corresponding to the one or more actions,
wherein the action advancement data corresponds to a type of data that is employed by the computing device and/or another computing device when performing a subtask of the given action of the one or more actions, and wherein the given action includes the subtask and other remaining subtasks;

causing the subtask of the given action of the one or more actions to be performed prior to the user providing the subsequent input to request performance of the given action of the one or more actions, wherein the subtask is in addition to any caching of the action advancement data;
determining, subsequent to determining that the user has provided the one or more inputs directed to the automated assistant, whether the user provided another input in furtherance of the computing device performing the given action of the one or more actions; and
when the user is determined to have provided the other input in furtherance of the given action:
causing, in response to determining the other input was provided by the user, the other remaining subtasks of the given action of the one or more actions to be performed such that performance of the subtask of the action is bypassed based on the subtask being previously performed.

12. The method of claim 11 , further comprising:
when the user is determined to have provided the other input in furtherance of the given action:
causing, in response to determining the other input was provided by the user, the trained machine learning model to be modified based on the user providing the other input that caused the given action to be performed.

13. The method of claim 11 , further comprising:
when the user is determined to have provided a particular input in furtherance of another action that is different from the given action:
causing, in response to determining the particular input was provided by the user

Assignees

• Google Llc

Inventors

• Mirelmann Lucas
• Sabur Zaheed
• Vlasyuk Bohdan
• Patriarche Bledowski Marie
• Nazarov Sergey
• Burakov Denis
• Behzadi Behshad
• Golikov Michael
• Cheng Steve
• Cotting Daniel
• Bertschler Mario

Classifications

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L2015/228

  of application context · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/083

  Recognition networks (G10L15/142, G10L15/16 take precedence) · CPC title