Automated assistant for generating, in response to a request from a user, application input content using application data from other sources

Patent metadata

Field	Value
Publication number	US-11972307-B2
Application number	US-201917049696-A
Country	US
Kind code	B2
Filing date	Aug 6, 2019
Priority date	May 6, 2019
Publication date	Apr 30, 2024
Grant date	Apr 30, 2024

Abstract

Implementations set forth herein relate to an automated assistant that can be invoked while a user is interfacing with a foreground application in order to retrieve data from one or more different applications, and then provide the retrieved data to the foreground application. A user can invoke the automated assistant while operating the foreground application by providing a spoken utterance, and the automated assistant can select one or more other applications to query based on content of the spoken utterance. Application data collected by the automated assistant from the one or more other applications can then be used to provide an input to the foreground application. In this way, the user can bypass switching between applications in the foreground in order to retrieve data that has been generated by other applications.

First claim

We claim:

1. A method implemented by one or more processors, the method comprising:
receiving, by an automated assistant application of a computing device, audio data captured by one or more microphones of the computing device, wherein the audio data captures a spoken utterance of a user who is accessing a given application at the computing device, the given application being accessible separate from the automated assistant application;
processing, by the automated assistant application using an on-device speech recognition engine, the audio data to generate textual data that corresponds to the spoken utterance;
processing, by the automated assistant application using an on-device natural language understanding (NLU) engine, the textual data to generate NLU data that indicates the spoken utterance corresponds to a request for the automated assistant application to interact with the given application using content that is provided by one or more other applications that are different from the given application and different from the automated assistant application;
in response to the NLU data indicating that the spoken utterance corresponds to a request for the automated assistant application to interact with the given application using the content that is provided by the one or more other applications:
generating, based on the textual data and/or the NLU data, an application data request for retrieving the content from the one or more other applications;
providing the application data request to the one or more other applications, wherein the one or more other applications are a subset of a group of applications installed on the computing device, and wherein the one or more other applications include a first application and a second application that are accessible separate from the automated assistant application and the given application,
receiving, from one or more of the other applications, corresponding responsive data that is responsive to the application data request,
generating, by the automated assistant application and based on the responsive data from one or more other applications, content for interacting with the given application, and
providing, by the automated assistant application to the given application, an application command that includes the content and that causes the given application to perform one or more actions based on the content.

2. The method of claim 1 , wherein the one or more actions include incorporating the content into one or more editable fields rendered at a graphical user interface of the given application.

3. The method of claim 2 , further comprising:
receiving, subsequent to providing the application command to the given application, a user input for modifying a portion of field content incorporated into an input field rendered at a graphical user interface of the given application,
wherein the portion of the field content modified by the user input corresponds to a portion of application data provided by multiple different applications of the one or more other applications.

4. The method of claim 1 , wherein the audio data captures the spoken utterance being received simultaneous to the user accessing a graphical user interface being rendered in a foreground of a display panel that is connected to the computing device.

5. The method of claim 1 ,
wherein the content is based on first application data that is provided by the first application and is formatted according to a data organizing protocol, and
wherein the content is also based on second application data that is provided by the second application and is not formatted according to the data organizing protocol.

6. The method of claim 1 , wherein generating the content for interacting with the given application is further based on assistant data that characterizes a previous interaction between the user and the automated assistant, and/or based on contextual data that characterizes a current context in which the user provided the spoken utterance to the automated assistant.

7. The method of claim 1 , wherein the given application is provided by an entity that is different from another entity that provided the automated assistant, or the given application is provided by a third party entity that is different from a separate entity that provided the automated assistant.

8. The method of claim 1 , further comprising:
selecting the one or more other applications based on the one or more applications each being assigned, in storage locally at the computing device, to an application type determined to correspond to the textual data and/or the NLU data;
wherein providing the application data request to the one or more other applications is responsive to selecting the one or more other applications.

9. The method of claim 1 , wherein providing the application data request to the one or more other applications comprises providing the application data request to an operating system of the computing device along with one or more identifiers of the one or more other applications, and wherein the operating system of the computing device directly provides the application data request to each of the one or more other applications based on the one or more identifiers.

10. The method of claim 1 , wherein receiving, from one or more of the other applications, corresponding responsive data that is responsive to the application data request comprises receiving responsive data from only a subgroup of the other applications.

11. A method implemented by one or more processors, the method comprising:
receiving, via an automated assistant interface of a computing device, a spoken utterance from a user who is accessing an application via the computing device,
wherein the spoken utterance corresponds to a request from the user for an automated assistant to insert field content into one or more application fields being rendered at a graphical user interface of the application;

accessing, in response to receiving the spoken utterance via the automated assistant interface, application data in furtherance of retrieving field content for the one or more application fields,
wherein the application data is provided by one or more other applications of the computing device, including by a first application of the one or more applications and a second application, that is different from the first application, of the one or more applications;

generating, based on accessing the application data, responsive data for providing the content for the one or more application fields,
wherein generating the responsive data includes retrieving data of the application data that is associated with application content being rendered at the graphical user interface of the application and natural language content embodied by the spoken utterance;

providing, based on generating the responsive data, the field content into one or more fields of the application; and
causing, subsequent to providing the field content into the one or more fields of the application, the application to perform one or more actions using the field content.

12. The method of claim 11 ,
wherein the application data includes first application data that is provided by the first application of the one or more applications and is formatted according to a data organizing protocol, and
wherein the application data includes second application data that is provided by the second application of the one or more applications and is not formatted according to the data organizing protocol.

13. The method of claim 11 , wherein the one or more fields are editable and provide the user with access to the field content when the fi

Assignees

• Google Llc

Inventors

• Vlasyuk Bohdan
• Behzadi Behshad
• Bertschler Mario
• Burakov Denis
• Cotting Daniel
• Golikov Michael
• Mirelmann Lucas
• Cheng Steve
• Nazarov Sergey
• Sabur Zaheed
• Lee Jonathan
• Terrenghi Lucia
• Zumbrunnen Adrian

Classifications

• G06F16/90332

  Natural language query formulation or dialogue systems · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06F9/547 Primary

  Remote procedure calls [RPC]; Web services · CPC title