Performing subtask(s) for a predicted action in response to a separate user interaction with an automated assistant prior to performance of the predicted action

Patent metadata

Field	Value
Publication number	US-11996102-B2
Application number	US-202318202236-A
Country	US
Kind code	B2
Filing date	May 25, 2023
Priority date	May 6, 2019
Publication date	May 28, 2024
Grant date	May 28, 2024

Abstract

Implementations relate to receiving natural language input that requests an automated assistant to provide information and processing the natural language input to identify the requested information and to identify one or more predicted actions. Those implementations further cause a computing device, at which the natural language input is received, to render the requested information and the one or more predicted actions in response to the natural language input. Yet further, those implementations, in response to the user confirming a rendered predicted action, cause the automated assistant to initialize the predicted action.

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
receiving natural language input from a user, wherein the natural language input requests an automated assistant to provide information, and wherein the natural language input is received at a computing device that provides access to the automated assistant;
processing the natural language input to identify the requested information and to identify one or more predicted actions, wherein processing the natural language input to identify the one or more predicted actions includes:
accessing application data that characterizes one or more features of an application that is executing at the computing device, and
processing the natural language input and the application data, using a machine learning model, to identify the one or more predicted actions;

causing the computing device to render, to the user and in response to the natural language input:
the requested information, and
the one or more predicted actions; and

in response to the user confirming a predicted action out of the one or more predicted actions, causing the automated assistant to initialize the predicted action.

2. The method of claim 1 , wherein the one or more features of the application that are characterized by the application data include content of a graphical interface being rendered at the direction of the application.

3. The method of claim 1 , wherein the predicted action is for a messaging application.

4. The method of claim 1 , further comprising:
prior to the user confirming the predicted action out of the one or more predicted actions, performing a subtask of the predicted action in response to receiving the natural language input, wherein causing the automated assistant to initialize the predicted action includes:
performing one or more remaining subtasks of the predicted action in response to the user confirming the predicted action.

5. The method of claim 4 , wherein the one or more subtasks include generating a request to be transmitted to perform the predicted action, obtaining network data for establishing a connection between the computing device and a third-party device for performing the predicted action, or communicating with the third-party device.

6. The method of claim 5 , wherein the one or more subtasks include generating the request to be transmitted to perform the predicted action.

7. The method of claim 1 , wherein, for each predicted action out of the one or more predicted actions, the machine learning model is used to generate a corresponding probability that the user will request performance of the corresponding predicted action, and further comprising:
ranking the one or more predicted actions based on the corresponding probability for each predicted action.

8. A method implemented by one or more processors, the method comprising:
receiving natural language input from a user, wherein the natural language input requests an automated assistant to provide information, and wherein the natural language input is received at a computing device that provides access to the automated assistant;
processing the natural language input to identify the requested information and to identify one or more predicted actions, wherein processing the natural language input to identify the one or more predicted actions includes:
identifying, based on the natural language input being from the user, contextual data that is specific to the user, and
processing the natural language input and the contextual data that is specific to the user, using a machine learning model, to identify the one or more predicted actions;

causing the computing device to render, to the user and in response to the natural language input:
the requested information, and
the one or more predicted actions; and

in response to the user confirming a predicted action out of the one or more predicted actions, causing the automated assistant to initialize the predicted action.

9. The method of claim 8 , wherein identifying the contextual data that is specific to the user is based on a frequency of utilization, by the user, associated with the contextual data.

10. The method of claim 8 , wherein the predicted action is for a messaging application.

11. The method of claim 8 , further comprising:
prior to the user confirming the predicted action out of the one or more predicted actions, performing a subtask of the predicted action in response to receiving the natural language input, wherein causing the automated assistant to initialize the predicted action includes:
performing one or more remaining subtasks of the predicted action in response to the user confirming the predicted action.

12. The method of claim 11 , wherein the one or more subtasks include generating a request to be transmitted to perform the predicted action, obtaining network data for establishing a connection between the computing device and a third-party device for performing the predicted action, or communicating with the third-party device.

13. The method of claim 12 , wherein the one or more subtasks include generating the request to be transmitted to perform the predicted action.

14. The method of claim 8 , wherein, for each predicted action out of the one or more predicted actions, the machine learning model is used to generate a corresponding probability that the user will request performance of the corresponding predicted action, and further comprising:
ranking the one or more predicted actions based on the corresponding probability for each predicted action.

15. The method of claim 8 , wherein the predicted action is performed by a third-party application.

16. A system comprising one or more processors and memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving natural language input from a user, wherein the natural language input requests an automated assistant to provide information, and wherein the natural language input is received at a computing device that provides access to the automated assistant;
processing the natural language input to identify the requested information and to identify one or more predicted actions, wherein processing the natural language input to identify the one or more predicted actions includes:
accessing application data that characterizes one or more features of an application that is executing at the computing device, and
processing the natural language input and the application data, using a machine learning model, to identify the one or more predicted actions;
causing the computing device to render, to the user and in response to the natural language input:
the requested information, and
the one or more predicted actions; and

in response to the user confirming a predicted action out of the one or more predicted actions, causing the automated assistant to initialize the predicted action.

17. The system of claim 16 , wherein the one or more features of the application that are characterized by the application data include content of a graphical interface being rendered at the direction of the application.

18. The system of claim 16 , wherein the predicted action is for a messaging application.

19. The system of claim 16 , wherein the predicted action is performed by a third-party application.

20. The system of claim 16 , wherein the operations performed by one or more of the processors when the instructions are executed further comprise:
prior

Assignees

• Google Llc

Inventors

• Mirelmann Lucas
• Sabur Zaheed
• Vlasyuk Bohdan
• Patriarche Bledowski Marie
• Nazarov Sergey
• Burakov Denis
• Behzadi Behshad
• Golikov Michael
• Cheng Steve
• Cotting Daniel
• Bertschler Mario

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/083

  Recognition networks (G10L15/142, G10L15/16 take precedence) · CPC title

• G10L2015/088

  Word spotting · CPC title

• G10L15/1815

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title