Dynamically delaying execution of automated assistant actions and/or background application requests

Patent metadata

Field	Value
Publication number	US-11545151-B2
Application number	US-201917045273-A
Country	US
Kind code	B2
Filing date	Jun 5, 2019
Priority date	Apr 26, 2019
Publication date	Jan 3, 2023
Grant date	Jan 3, 2023

Abstract

Implementations set forth herein allow a user to access a first application in a foreground of a graphical interface, and simultaneously employ an automated assistant to respond to notifications arising from a second application. The user can provide an input, such as a spoken utterance, while viewing the first application in the foreground in order to respond to notifications from the second application without performing certain intervening steps that can arise under certain circumstances. Such intervening steps can include providing a user confirmation, which can be bypassed, and/or time-limited according to a timer, which can be displayed in response to the user providing a responsive input directed at the notification. A period for the timer can be set according to one or more characteristics that are associated with the notification, the user, and/or any other information that can be associated with the user receiving the notification.

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
determining, at a computing device, that a user provided a spoken utterance to an interface of the computing device at which a first application and a second application are executing,
wherein the first application is being rendered in foreground of a graphical user interface that is accessible via a display panel of the computing device, and
wherein the computing device provides access to an automated assistant that is accessible separate from the first application and the second application;

determining, based on the user providing the spoken utterance and when the first application is rendered in the foreground of the graphical user interface, whether the spoken utterance is directed to the first application and/or the second application; and
when the spoken utterance is determined to be directed to the second application:
generating, based on natural language content of the spoken utterance:
an action request directed to the second application, and
a confidence metric for predicting whether the action request will be confirmed and/or modified before the action request is provided to the second application;

determining, based on the confidence metric, whether to render an action-impending notification in the foreground of the graphical user interface,
wherein the action-impending notification includes a graphical element that characterizes a duration of a delay before the action request is provided to the second application;

wherein, when the confidence metric indicates that the action-impending notification should be rendered, the method further comprises:
causing the action-impending notification to be rendered in the foreground of the graphical user interface; and

wherein, when the confidence metric indicates that the action-impending notification should not be rendered, the method further comprises:
bypassing causing the action-impending notification to be rendered in the foreground of the graphical user interface; and
causing the action request to be provided to the second application via the automated assistant.

2. The method of claim 1 ,
wherein the duration of the delay before the action request is provided to the second application is based on the confidence metric, and
wherein causing the action-impending notification to be rendered in the foreground of the graphical user interface includes:
causing the action-impending notification to be rendered in the foreground for the duration of the delay or until the user acts to confirm and/or modify the action request.

3. The method as in claim 1 , wherein generating the confidence metric includes:
processing input data that characterizes one or more features of a historical interaction between the user and the automated assistant, the user and the first application, and/or the user and the second application.

4. The method of claim 1 , further comprising:
when the confidence metric indicates that the action-impending notification should be rendered:
determining that the user confirmed the action-impending notification rendered in the foreground of the graphical user interface within the duration; and
causing, in response to determining that the user confirmed the action-impending notification, the action request to be provided to the second application via the automated assistant.

5. The method of claim 1 , further comprising:
when the confidence metric indicates that the action-impending notification should be rendered:
determining that the user did not confirm the action-impending notification rendered in the foreground of the graphical user interface within the duration; and
causing, in response to determining that the user did not confirm the action-impending notification, a trained machine learning model to be modified based on the user not confirming the action-impending notification,
wherein the trained machine learning model is used in generating the confidence metric.

6. The method of claim 1 , wherein determining whether the spoken utterance is directed to the first application and/or the second application includes:
identifying a particular application that provided a most recent notification, wherein the most recent notification corresponds to an incoming message from a sender, and wherein the second application is the particular application.

7. A computing device, comprising:
a display panel;
memory storing instructions;
one or more processors operable to execute the instructions to:
determine that a user provided a spoken utterance to an interface of the computing device, wherein a first application and a second application are executing at the computing device,
wherein the first application is being rendered in foreground of a graphical user interface that is accessible via the display panel of the computing device, and
wherein the computing device provides access to an automated assistant that is accessible separate from the first application and the second application;

determine, based on the user providing the spoken utterance and when the first application is rendered in the foreground of the graphical user interface, whether the spoken utterance is directed to the first application and/or the second application; and
when the spoken utterance is determined to be directed to the second application:
generate, based on natural language content of the spoken utterance:
an action request directed to the second application, and
a confidence metric for predicting whether the action request will be confirmed and/or modified before the action request is provided to the second application;

determine, based on the confidence metric, whether to render an action-impending notification in the foreground of the graphical user interface,
wherein the action-impending notification includes a graphical element that characterizes a duration of a delay before the action request is provided to the second application;

wherein, when the confidence metric indicates that the action-impending notification should be rendered, the one or more processors:
cause the action-impending notification to be rendered in the foreground of the graphical user interface; and

wherein, when the confidence metric indicates that the action-impending notification should not be rendered, the one or more processors:
bypass causing the action-impending notification to be rendered in the foreground of the graphical user interface; and
cause the action request to be provided to the second application via the automated assistant.

8. The computing device of claim 7 ,
wherein the duration of the delay before the action request is provided to the second application is based on the confidence metric, and
wherein in causing the action-impending notification to be rendered in the foreground of the graphical user interface one or more of the processors are to:
cause the action-impending notification to be rendered in the foreground for the duration of the delay or until the user acts to confirm and/or modify the action request.

9. The computing device of claim 7 , wherein in generating the confidence metric one or more of the processors are to:
process input data that characterizes one or more features of a historical interaction between the user and the automated assistant, the user and the first application, and/or the user and the second application.

10. The computing device of claim 7 , wherein one or more of the processors, in executing the instructions, are further to:
when the confidence metric indicates that the action-impending notification should be rendered:
determine that the user confirmed the action-impending noti

Assignees

• Google Llc

Inventors

• Burakov Denis
• Nazarov Sergey
• Behzadi Behshad
• Bertschler Mario
• Vlasyuk Bohdan
• Cotting Daniel
• Golikov Michael
• Mirelmann Lucas
• Cheng Steve
• Sabur Zaheed
• Kolak Okan
• Zhong Yan
• Ly Vinh Quoc

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L2015/227

  of the speaker; Human-factor methodology · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L2015/228

  of application context · CPC title