Intelligent automated assistant in a messaging environment

Patent metadata

Field	Value
Publication number	US-11526368-B2
Application number	US-202015931384-A
Country	US
Kind code	B2
Filing date	May 13, 2020
Priority date	Nov 6, 2015
Publication date	Dec 13, 2022
Grant date	Dec 13, 2022

Abstract

Systems and processes for operating an intelligent automated assistant in a messaging environment are provided. In one example process, a graphical user interface (GUI) having a plurality of previous messages between a user of the electronic device and the digital assistant can be displayed on a display. The plurality of previous messages can be presented in a conversational view. User input can be received and in response to receiving the user input, the user input can be displayed as a first message in the GUI. A contextual state of the electronic device corresponding to the displayed user input can be stored. The process can cause an action to be performed in accordance with a user intent derived from the user input. A response based on the action can be displayed as a second message in the GUI.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with a display, cause the electronic device to:
display, on the display, a graphical user interface (GUI) having a plurality of previous messages between a user and a digital assistant, the plurality of previous messages presented in a conversational view;
receive a first user input including a media object;
in response to receiving the first user input, display the media object as a first message in the GUI;
receive a second user input including text;
in response to receiving the second user input, display the text as a second message in the GUI;
cause a user intent to be determined based on a combination of the media object and the text; and
after the user intent is determined:
obtain a determination of whether the user intent requires extracting text from the media object; and
in response to obtaining a determination that the user intent requires extracting text from the media object:
extract text from the media object;
perform, using the extracted text, a task in accordance with the user intent; and
display, as a third message in the GUI, a response indicative of the user intent being satisfied.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
in accordance with the user intent, populate the extracted text into a text field of an application of the electronic device.

3. The non-transitory computer-readable storage medium of claim 1 , wherein the user intent comprises creating, using the media object, a contact entry in a contacts application of the electronic device.

4. The non-transitory computer-readable storage medium of claim 3 , wherein:
the media object is an image depicting contact information of an entity;
the extracted text includes the contact information; and
performing the task in accordance with the user intent further comprises populating a text field of the contact entry with the extracted text, the contact entry associated with the entity.

5. The non-transitory computer-readable storage medium of claim 1 , wherein the user intent comprises creating, using the media object, a calendar entry in a calendar application of the electronic device.

6. The non-transitory computer-readable storage medium of claim 5 , wherein:
the media object is an image depicting event information;
the extracted text includes the event information; and
performing the task in accordance with the user intent further comprises populating a text field of the calendar entry with the extracted text.

7. The non-transitory computer-readable storage medium of claim 1 , wherein the user intent comprises creating, using the media object, a reminder entry in a reminder application of the electronic device.

8. The non-transitory computer-readable storage medium of claim 7 , wherein:
the media object is an image depicting a reminder task;
the extracted text includes the reminder task; and
performing the task in accordance with the user intent further comprises populating a text field of the reminder entry with the extracted text.

9. The non-transitory computer-readable storage medium of claim 1 , wherein the user intent comprises translating text of a first language in the media object to text of a second language.

10. The non-transitory computer-readable storage medium of claim 9 , wherein:
the media object is an image depicting the text of the first language;
the extracted text includes the text of the first language;
performing the task in accordance with the user intent further comprises obtaining the text of the second language corresponding to the text of the first language; and
the displayed response includes the text of the second language.

11. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
in response to obtaining a determination that the user intent does not require extracting text from the media object, obtain a determination of whether the user intent requires performing image recognition on the media object; and
in response to obtaining a determination that the user intent requires performing image recognition on the media object:
cause image recognition on the media object to be performed;
obtain, based on the image recognition, information associated with the media object; and
display, as a fourth message in the GUI, a response indicative of the user intent being satisfied, the response based on the information associated with the media object.

12. The non-transitory computer-readable storage medium of claim 11 , wherein the media object depicts a retail object, and wherein the information associated with the media object includes price information of the retail object.

13. The non-transitory computer-readable storage medium of claim 11 , wherein the media object depicts a location, and wherein the information associated with the media object includes an identity of the location.

14. The non-transitory computer-readable storage medium of claim 11 , wherein the media object depicts an entity, and wherein the information associated with the media object includes an identity of the entity.

15. The non-transitory computer-readable storage medium of claim 11 , wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
in response to obtaining a determination that the user intent does not require performing image recognition on the media object, obtain a determination of whether the user intent requires performing audio processing on the media object; and
in response to obtaining a determination that the user intent requires performing audio processing on the media object:
cause audio processing on the media object to be performed;
obtain, based on the audio processing, information associated with the media object; and
display, as a fifth message in the GUI, a response indicative of the user intent being satisfied, the response based on the information associated with the media object.

16. The non-transitory computer-readable storage medium of claim 15 , wherein causing audio processing on the media object to be performed further comprises:
causing speech-to-text recognition to be performed on the media object to obtain text corresponding to speech in the media object.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the information is obtained using the text corresponding to the speech in the media object.

18. The non-transitory computer-readable storage medium of claim 16 , wherein the text corresponding to the speech in the media object is stored in association with an application of the electronic device in accordance with the user intent.

19. The non-transitory computer-readable storage medium of claim 15 , wherein causing audio processing on the media object to be performed further comprises:
causing audio recognition to be performed using the media object to obtain text identifying the media object.

20. The non-transitory comput

Assignees

• Apple Inc

Inventors

• KARASHCHUK Petr
• Vega Galvez Tomas A
• Gruber Thomas R

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• H04L51/10

  Multimedia information · CPC title

• G06Q10/107

  Computer-aided management of electronic mailing [e-mailing] · CPC title

• H04W4/12

  Messaging; Mailboxes; Announcements · CPC title