Performing subtask(s) for a predicted action in response to a separate user interaction with an automated assistant prior to performance of the predicted action

Patent metadata

Field	Value
Publication number	US-11664028-B2
Application number	US-202217569811-A
Country	US
Kind code	B2
Filing date	Jan 6, 2022
Priority date	May 6, 2019
Publication date	May 30, 2023
Grant date	May 30, 2023

Abstract

Implementations herein relate to pre-caching data, corresponding to predicted interactions between a user and an automated assistant, using data characterizing previous interactions between the user and the automated assistant. An interaction can be predicted based on details of a current interaction between the user and an automated assistant. One or more predicted interactions can be initialized, and/or any corresponding data pre-cached, prior to the user commanding the automated assistant in furtherance of the predicted interaction. Interaction predictions can be generated using a user-parameterized machine learning model, which can be used when processing input(s) that characterize a recent user interaction with the automated assistant. The predicted interaction(s) can include action(s) to be performed by third-party application(s).

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
receiving a spoken utterance from a user, wherein the spoken utterance is directed to an automated assistant and is received at a computing device that provides access to the automated assistant;
processing data characterizing the spoken utterance to identify a requested action requested by the user through the spoken utterance and to identify one or more action predictions, wherein processing the data characterizing the spoken utterance to identify the one or more action predictions includes:
accessing contextual data associated with the spoken utterance,
using a machine learning model to determine, based on the requested action and the contextual data associated with the spoken utterance, one or more predicted actions, and
identifying at least one third-party application for performing a first predicted action of the one or more predicted actions, the third-party application being accessible via the automated assistant; and

in response to receiving the spoken utterance, causing the computing device to render data based on the requested action and to perform an additional action based on the one or more predicted actions.

2. The method of claim 1 , wherein causing the computing device to perform the additional action includes:
generating action advancement data for one or more subtasks of the one or more predicted actions.

3. The method of claim 2 , wherein the one or more subtasks include generating a request to be transmitted to the third-party application, obtaining network data for establishing a connection between the computing device and a third-party device for performing another predicted action, or communicating with a third-party server.

4. The method of claim 2 , wherein causing the computing device to perform the additional action further includes:
estimating a computational obligation of a subtask of the one or more subtasks, and
caching the action advancement data for the subtask for a given amount of time, the given amount of time being determined based on the estimated computational obligation of the subtask.

5. The method of claim 2 , further comprising:
prior to receiving a subsequent user input that is associated with one or more of the predicted actions:
accessing the action advancement data, and
performing, using the action advancement data, one or more of the subtasks.

6. The method of claim 2 , wherein the action advancement data includes application data associated with the third-party application.

7. The method of claim 2 , wherein the action advancement data includes connection data and/or authentication data for connecting the computing device with another device.

8. The method of claim 2 , wherein the action advancement data includes content to be rendered to the user in response to the user providing a subsequent request for a predicted action.

9. The method of claim 1 , further comprising:
receiving a subsequent user confirmation for performing a second predicted action, the second predicted action being the same or different from the first predicted action, and
in response to receiving the subsequent user confirmation, causing the performance of the second predicted action, and
modifying the machine learning model based on the subsequent user confirmation.

10. The method of claim 9 , wherein in response to receiving the subsequent user input,
causing the performance of the second predicted action comprises:
determining whether the subsequent user input is received within a threshold period of time,
in response to determining that the subsequent user input is received within the threshold period of time, causing the second predicted action to be performed, and
in response to determining that the subsequent user input is not received within the threshold period of time, causing the second predicted action to be bypassed.

11. The method of claim 1 , further comprising:
receiving an additional user input within a threshold period, the additional user input not being associated with the first predicted action, and
causing the first predicted action to be bypassed.

12. The method of claim 1 , wherein the data based on the requested action reflects performance of the requested action.

13. The method of claim 1 , wherein, for each predicted action out of the one or more predicted actions, the machine learning model is used to generate a corresponding probability that the user will request performance of the corresponding predicted action, and further comprising:
ranking the one or more predicted actions based on the corresponding probability for each predicted action.

14. The method of claim 1 , further comprising:
estimating a computational obligation for each of the one or more predicted actions, and
prioritizing the one or more predicted actions based on the computational obligation determined for each of the one or more predicted actions.

15. The method of claim 1 , wherein the machine learning model is trained using historical interaction between the user and the automated assistant, and/or using historical interaction between one or more additional users and the automated assistant.

16. The method of claim 1 , wherein the contextual data includes: one or more pictures of a context in which the user provides the spoken utterance, one or more operating features of the computing device, and/or a location of the computing device.

17. The method of claim 1 , wherein the contextual data includes a context in which one or more applications are executing at the computer device.

18. A method implemented by one or more processors, the method comprising:
receiving a spoken utterance from a user, wherein the spoken utterance requests an automated assistant to provide information, and wherein the spoken utterance is received at a computing device that provides access to the automated assistant;
processing the spoken utterance to identify the requested information and to identify one or more predicted actions, wherein processing the spoken utterance to identify the one or more predicted actions includes:
accessing contextual data associated with the spoken utterance, and
processing the spoken utterance and the contextual data, using a machine learning model, to identify the one or more predicted actions;

causing the computing device or another device to render, to the user, the requested information; and
in response to the user confirming a predicted action out of the one or more predicted actions, causing the automated assistant to initialize the predicted action.

19. The method of claim 18 , further comprising:
prior to the user confirming the predicted action out of the one or more predicted actions, performing a subtask of the predicted action in responsive to the spoken utterance, wherein causing the automated assistant to initialize the predicted action includes:
performing remaining subtasks of the predicted action in responsive to the user confirming the predicted action.

20. A system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving a spoken utterance from a user, wherein the spoken utterance requests an automated assistant to provide information, and wherein the spoken utterance is received at a computing device that provides access to the automated assistant;
processing the spoken utterance to identify the requeste

Assignees

• Google Llc

Inventors

• Mirelmann Lucas
• Sabur Zaheed
• Vlasyuk Bohdan
• Patriarche Bledowski Marie
• Nazarov Sergey
• Burakov Denis
• Behzadi Behshad
• Golikov Michael
• Cheng Steve
• Cotting Daniel
• Bertschler Mario

Classifications

• G10L2015/228

  of application context · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L15/1815

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G10L15/083

  Recognition networks (G10L15/142, G10L15/16 take precedence) · CPC title