Feedback analysis of a digital assistant

Patent metadata

Field	Value
Publication number	US-11301477-B2
Application number	US-201715675637-A
Country	US
Kind code	B2
Filing date	Aug 11, 2017
Priority date	May 12, 2017
Publication date	Apr 12, 2022
Grant date	Apr 12, 2022

Abstract

Systems and processes for operating an intelligent automated assistant are provided. In accordance with one example, a method includes, at an electronic device with one or more processors and memory, obtaining a first set of data corresponding to one or more interactions between a user and the digital assistant on the electronic device; obtaining a second set of data corresponding to one or more interactions between the user and an application on the electronic device; and storing the first set of data and the second set of data. The method further includes receiving a set of executable instructions; processing, based on the received set of executable instructions, the first set of data and the second set of data to obtain a third set of data; and transmitting the third set of data to a remote electronic device.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:
obtain a first set of data corresponding to one or more interactions between a user of the electronic device and a digital assistant on the electronic device, wherein the first set of data includes multiple candidate text representations of a speech input provided to the digital assistant;
obtain a second set of data corresponding to one or more interactions between the user and an application on the electronic device, wherein the second set of data includes a user input provided to the application;
store, by the electronic device, the first set of data and the second set of data;
receive, by the electronic device, a set of executable instructions from a remote electronic device, wherein the set of executable instructions include instructions for comparing the multiple candidate text representations to the user input to determine that the user input is a modified version of a candidate text representation of the multiple candidate text representations;
process based on the received set of executable instructions, by the electronic device, the first set of data and the second set of data to obtain a third set of data; and
transmit, by the electronic device, the third set of data to the remote electronic device.

2. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more interactions between the user and the digital assistant on the electronic device include the speech input, and wherein obtaining the first set of data comprises obtaining data corresponding to the speech input.

3. The non-transitory computer-readable storage medium of claim 1 , wherein obtaining the first set of data comprises obtaining data corresponding to an output provided by the digital assistant.

4. The non-transitory computer-readable storage medium of claim 1 , wherein obtaining the first set of data comprises obtaining data corresponding to one or more events associated with a session of the digital assistant.

5. The non-transitory computer-readable storage medium of claim 4 , wherein obtaining the first set of data comprises obtaining timing information corresponding to at least one event of the one or more events.

6. The non-transitory computer-readable storage medium of claim 1 , wherein the second set of data includes data corresponding to at least one of:
creation of a data item,
deletion of a data item,
modifications to a data item,
transmission of a data item,
start of an operation,
cancellation of an operation,
completion of an operation, or
any combination thereof.

7. The non-transitory computer-readable storage medium of claim 1 , wherein the second set of data includes data corresponding to a user intent.

8. The non-transitory computer-readable storage medium of claim 1 , wherein the application is at least one of:
a calendar application,
a reminder application,
a timer application,
an alarm application,
a map application,
a messaging application,
a shopping application,
a music application,
a web browser, or
any combination thereof.

9. The non-transitory computer-readable storage medium of claim 1 , wherein the application is associated with a domain of the digital assistant.

10. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:
determine, by the electronic device, whether a set of conditions of the electronic device is met;
in accordance with a determination that the set of conditions is met, send, to the remote electronic device, a request for the set of executable instructions; and
in accordance with a determination that the set of conditions is not met, forego sending, to the remote electronic device, the request for the set of executable instructions.

11. The non-transitory computer-readable storage medium of claim 1 , wherein processing the first set of data and the second set of data to obtain the third set of data comprises:
determining, based on the set of executable instructions, that a portion of the first set of data and a portion of the second set of data are associated with a same request to the digital assistant.

12. The non-transitory computer-readable storage medium of claim 1 , wherein processing the first set of data and the second set of data to obtain the third set of data comprises:
anonymizing, at the electronic device, the first set of data or the second set of data.

13. The non-transitory computer-readable storage medium of claim 12 , wherein anonymizing the first set of data or the second set of data comprises:
determining, at the electronic device, that a portion of the first set of data or the second set of data corresponds to a predefined type of data, and
excluding the portion of the first set of data or the second set of data from the third set of data.

14. The non-transitory computer-readable storage medium of claim 13 , wherein processing the first set of data and the second set of data to obtain a third set of data comprises:
comparing, at the electronic device, a portion of the first set of data with a portion of the second set of data; and
identifying, based on the comparison, a first result from the first set of data and a second result from the second set of data,
wherein the first result is obtained by the digital assistant, and
wherein the second result is a user-altered version of the first result.

15. The non-transitory computer-readable storage medium of claim 1 , wherein processing the first set of data and the second set of data to obtain a third set of data comprises:
aggregating a portion of the first set of data and a portion of the second set of data.

16. The non-transitory computer-readable storage medium of claim 1 ,
wherein the remote electronic device:
determines whether the third set of data is indicative of a user-altered performance of a task;
in accordance with a determination that the third set of data is indicative of a user-altered performance of a task, updates one or more components associated with the digital assistant; and
in accordance with a determination that the third set of data is not indicative of a user-altered performance of a task, foregoes updating the one or more components associated with the digital assistant.

17. The non-transitory computer-readable storage medium of claim 16 , wherein updating one or more components associated with the digital assistant comprises creating a target test set based on the third set of data.

18. The non-transitory computer-readable storage medium of claim 16 , wherein updating one or more components associated with the digital assistant comprises updating a ranking model associated with the digital assistant.

19. The non-transitory computer-readable storage medium of claim 1 , wherein the electronic device is a computer, a set-top box, a speaker, a smart watch, a phone, or any combination thereof.

20. The non-transitory computer-readable storage medium of claim 1 , wherein the set of executable instructions specify to exclude a portion of the first set of data and the second set of data from the third set of data.

21. The non-transitory computer-readable storage medium of c

Assignees

• Apple Inc

Inventors

• Frazzingaro Gennaro
• Garg Ashish
• Thomson Blaise
• Kumar Anuj
• Shah Rushin N
• Seigel Matthew S
• Li Lin

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F16/243

  Natural language query formulation · CPC title

• G06F21/6254

  by anonymising data, e.g. decorrelating personal data from the owner's identification · CPC title

• G06F16/24564 Primary

  Applying rules; Deductive queries · CPC title