Dynamically delaying execution of automated assistant actions and/or background application requests

Patent metadata

Field	Value
Publication number	US-11749280-B2
Application number	US-202218086263-A
Country	US
Kind code	B2
Filing date	Dec 21, 2022
Priority date	Apr 26, 2019
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

Implementations set forth herein allow a user to access a first application in a foreground of a graphical interface, and simultaneously employ an automated assistant to respond to notifications arising from a second application. The user can provide an input, such as a spoken utterance, while viewing the first application in the foreground in order to respond to notifications from the second application without performing certain intervening steps that can arise under certain circumstances. Such intervening steps can include providing a user confirmation, which can be bypassed, and/or time-limited according to a timer, which can be displayed in response to the user providing a responsive input directed at the notification. A period for the timer can be set according to one or more characteristics that are associated with the notification, the user, and/or any other information that can be associated with the user receiving the notification.

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
determining that a user provided a spoken utterance to an interface of a computing device, wherein the computing device provides access to an automated assistant via the interface;
determining, based on processing the spoken utterance, that the spoken utterance is directed to an action for an application to perform, wherein the application is separate from the automated assistant;
determining, based on the spoken utterance, a duration of a delay before initializing the action via an action request provided by the automated assistant;
causing an action impending notification to be rendered, in a foreground of a graphical user interface for the duration of the delay, wherein the action-impending notification comprises:
a characterization of content of the action request, and
a dynamic graphical representation of a remaining duration of the duration of the delay;

monitoring, during the duration of the delay, for confirmation input from the user for confirming the action or modification input from the user for modifying the action;
when, based on the monitoring, the user is determined to have provided the modification input for modifying the action during the duration of the delay:
causing the automated assistant to at least temporarily bypass initializing the action; and

when, based on the monitoring, the user is determined to have provided the confirmation input for confirming the action request during the duration of the delay or is determined to have allowed the duration of the delay to expire without providing either the confirmation input or the modification input:
causing the automated assistant to initialize performance of the action by providing the action request.

2. The method of claim 1 , further comprising:
accessing contextual data characterizing one or more properties of a context in which the user provided the spoken utterance, wherein determining the duration of the delay is further based on the contextual data.

3. The method of claim 1 , wherein determining, based on the spoken utterance, the duration of the delay comprises:
generating, based on natural language content of the spoken utterance, a confidence metric; and
determining the duration of the delay based on the confidence metric.

4. The method of claim 3 , wherein generating the confidence metric is further based on input data that characterizes one or more features of a historical interaction between the user and the automated assistant.

5. The method of claim 1 , wherein determining the spoken utterance is directed to the action for the application to perform further comprises:
determining the application based on the application having provided a most recent notification, wherein the most recent notification corresponds to an incoming message from a sender.

6. The method of claim 5 , wherein determining the duration of the delay is further based on a sender of the incoming message.

7. The method of claim 1 , wherein determining, based on the spoken utterance, the duration of the delay comprises:
determining the duration of the delay based on a quantity of words in the spoken utterance and/or a quantity of characters in the words.

8. The method of claim 1 , wherein the action includes sending a message, wherein natural language content of the spoken utterance includes words to include in the message, and wherein determining, based on the spoken utterance, the duration of the delay comprises:
determining the duration of the delay based on a quantity of characters in the words and/or a quantity of the words.

9. The method of claim 1 , wherein determining, based on the spoken utterance, the duration of the delay comprises:
determining one or more confidence metrics based on speech to text conversion of the spoken utterance; and
determining the duration of the delay is based on the one or more confidence metrics.

10. The method of claim 1 , wherein determining, based on the spoken utterance, the duration of the delay comprises:
determining, based on historical user data, a level of similarity between the action and previous user actions; and
determining the duration of the delay based on the level of similarity.

11. A method implemented by one or more processors, the method comprising:
determining that a user provided a spoken utterance to an interface of a computing device, wherein the computing device provides access to an automated assistant via the interface;
determining, based on processing the spoken utterance, that the spoken utterance is directed to an action for an application to perform, wherein the application is separate from the automated assistant;
determining, based on the spoken utterance, to render an action impending notification in the foreground of a graphical user interface for a duration of a delay before initializing the action via an action request provided by the automated assistant, wherein the action-impending notification comprises:
a characterization of content of the action request, and
a dynamic graphical representation of a remaining duration of the duration of the delay;

monitoring, during the duration of the delay, for confirmation input from the user for confirming the action or modification input from the user for modifying the action;
when, based on the monitoring, the user is determined to have provided the modification input for modifying the action during the duration of the delay:
causing the automated assistant to at least temporarily bypass initializing the action; and

when, based on the monitoring, the user is determined to have provided the confirmation input for confirming the action request during the duration of the delay or is determined to have allowed the duration of the delay to expire without providing either the confirmation input or the modification input:
causing the automated assistant to initialize performance of the action by providing the action request.

12. The method of claim 11 , wherein the action comprises sending a message to an additional user of an additional computing device.

13. The method of claim 12 , further comprising:
determining, based on the spoken utterance, the duration of the delay.

14. The method of claim 13 , wherein the message is determined based on the spoken utterance and wherein determining the duration of the delay based on the spoken utterance comprises:
determining the duration of the delay based on a length of the message.

15. A method implemented by one or more processors, the method comprising:
determining that a user provided a spoken utterance to an interface of a computing device, wherein the computing device provides access to an automated assistant via the interface;
determining, based on processing the spoken utterance, that the spoken utterance is directed to an action for an application to perform, wherein the application is separate from the automated assistant;
determining, based on the action, a duration of a delay before initializing the action via an action request provided by the automated assistant;
causing an action impending notification to be rendered, in a foreground of a graphical user interface for the duration of the delay, wherein the action-impending notification comprises:
a characterization of content of the action request, and
a dynamic graphical representation of a remaining duration of the duration of the delay;

monitoring, during the duration of the delay, for confirmation input from the user for confirming the action or modification input from the user for modifying the action;

Assignees

• Google Llc

Inventors

• Burakov Denis
• Nazarov Sergey
• Behzadi Behshad
• Bertschler Mario
• Vlasyuk Bohdan
• Cotting Daniel
• Golikov Michael
• Mirelmann Lucas
• Cheng Steve
• Sabur Zaheed
• Kolak Okan
• Zhong Yan
• Ly Vinh Quoc

Classifications

• G10L2015/228

  of application context · CPC title

• G10L2015/227

  of the speaker; Human-factor methodology · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title