Intelligent automated assistant in a messaging environment

Patent metadata

Field	Value
Publication number	US-10691473-B2
Application number	US-201615151191-A
Country	US
Kind code	B2
Filing date	May 10, 2016
Priority date	Nov 6, 2015
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

Systems and processes for operating an intelligent automated assistant in a messaging environment are provided. In one example process, a graphical user interface (GUI) having a plurality of previous messages between a user of the electronic device and the digital assistant can be displayed on a display. The plurality of previous messages can be presented in a conversational view. User input can be received and in response to receiving the user input, the user input can be displayed as a first message in the GUI. A contextual state of the electronic device corresponding to the displayed user input can be stored. The process can cause an action to be performed in accordance with a user intent derived from the user input. A response based on the action can be displayed as a second message in the GUI.

First claim

What is claimed is:

1. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors of an electronic device having a display, cause the one or more processors to:
display, on the display, a graphical user interface (GUI) having a plurality of previous messages between a user of the electronic device and a digital assistant implemented on the electronic device, the plurality of previous messages presented in a conversational view;
after displaying the plurality of previous messages, detect a user selection of a first previous message of the displayed plurality of previous messages, the first previous message corresponding to a first previous user input received at a first time;
in response to detecting the user selection of the first previous message, retrieve a first previous contextual state of the electronic device at the first time, wherein the first previous contextual state is associated with the first previous message;
receive a current user input at a second time after the first time; and
in response to receiving the current user input:
display a representation of the current user input as a first current message in the GUI, wherein the first current message is associated with a current contextual state of the electronic device at the second time;
cause a determination of a user intent based on the current user input and the retrieved first previous contextual state of the electronic device at the first time;
cause an action to be performed in accordance with the determined user intent, wherein results are obtained by performing the action; and
display a response as a second current message in the GUI, the response containing a representation of the obtained results.

2. The non-transitory computer-readable medium of claim 1 , wherein each of the plurality of previous messages is associated with a respective previous contextual state of the electronic device.

3. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the one or more processors to:
prior to the second time:
detect a second previous user input;
in response to detecting the second previous user input, sample audio data containing a user utterance;
cause a second action to be performed in accordance with a second user intent derived from the user utterance; and
output a second response based on the second action, wherein a second previous message of the plurality of previous messages includes a text representation of the user utterance, and wherein a third previous message of the plurality of previous messages is based on the second response.

4. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the one or more processors to:
prior to the second time, receive a third previous user input associated with a fourth previous message of the plurality of previous messages;
in response to receiving the third previous user input, store a bookmark of the fourth previous message in association with an affordance; and
display the affordance on the GUI, wherein selecting the affordance causes a focus of the GUI to change to the fourth previous message.

5. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the one or more processors to:
in response to displaying the current user input as the first current message in the GUI of the electronic device, cause display of the first current message in a GUI of a second electronic device; and
in response to displaying the response as the second current message in the GUI of the electronic device, cause display of the second current message in the GUI of the second electronic device.

6. The non-transitory computer-readable medium of claim 5 , wherein the instructions further cause the one or more processors to cause the second electronic device to:
detect a user selection of the first current message in the GUI of the second electronic device;
receive a fourth user input at a third time after the second time;
in response to receiving the fourth user input, display the fourth user input as a third current message in the GUI of the second electronic device;
cause a third action to be performed in accordance with a third user intent, the third user intent derived from the fourth user input and the current contextual state of the electronic device at the second time; and
display a third response as a fourth current message in the GUI of the second electronic device, the third response based on the third action.

7. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the one or more processors to:
identify, based on the current contextual state of the electronic device and the determined user intent, one or more words in the displayed response corresponding to an entity;
determine a fourth action associated with the entity; and
enable selection of the one or more words in the displayed response, wherein detecting a user selection of the one or more words causes the fourth action associated with the entity to be performed.

8. The non-transitory computer-readable medium of claim 1 , wherein the GUI of the electronic device includes a text input field, and wherein the instructions further cause the one or more processors to:
after displaying the GUI, detect a fifth user input associated with a fifth previous message of the plurality of previous messages; and
in response to detecting the fifth user input, display text of the fifth previous message in the text input field of the GUI.

9. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the one or more processors to:
receive, via the GUI of the electronic device, a user selection of the second current message; in response to receiving the user selection of the second current message:
obtain, using an application of the electronic device, detailed results based on the user intent; and
display the detailed result in a GUI of the application.

10. The non-transitory computer-readable medium of claim 9 , wherein displaying the response as the second current message includes displaying an indication in the second current message that a more detailed response is available in response to selecting the second current message.

11. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the one or more processors to:
in response to receiving the current user input, display, based on the user intent, a media object in the GUI of the electronic device, the media object associated with an application of the electronic device.

12. The non-transitory computer-readable medium of claim 11 , wherein the instructions further cause the one or more processors to:
detect a user selection of the media object;
in response to detecting the user selection of the media object:
obtain, using the application of the electronic device, expanded results corresponding to the media object; and
display the expanded results in a GUI of the application.

13. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the one or more processors to:
receive a notification from an application of the electronic device; and
in response to receiving the notification, display the notification as a fifth current message in the GUI.

14. The non-transitory computer-readable medium of claim 1 , wherein the GUI of the electronic device includes a user input interface, wherein the displayed response includes a request for additio

Assignees

• Apple Inc

Inventors

• KARASHCHUK Petr
• Vega Galvez Tomas A
• Gruber Thomas R

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• H04W4/12

  Messaging; Mailboxes; Announcements · CPC title

• H04L51/046

  Interoperability with other network applications or services · CPC title

• H04L51/02 Primary

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title