Updating language understanding classifier models for a digital personal assistant based on crowd-sourcing

Patent metadata

Field	Value
Publication number	US-9508339-B2
Application number	US-201514611042-A
Country	US
Kind code	B2
Filing date	Jan 30, 2015
Priority date	Jan 30, 2015
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

A method for updating language understanding classifier models includes receiving via one or more microphones of a computing device, a digital voice input from a user of the computing device. Natural language processing using the digital voice input is used to determine a user voice request. Upon determining the user voice request does not match at least one of a plurality of pre-defined voice commands in a schema definition of a digital personal assistant, a GUI of an end-user labeling tool is used to receive a user selection of at least one of the following: at least one intent of a plurality of available intents and/or at least one slot for the at least one intent. A labeled data set is generated by pairing the user voice request and the user selection, and is used to update a language understanding classifier.

First claim

What is claimed is:

1. A server computer, comprising:
a processing unit; and
memory coupled to the processing unit;
the server computer configured to perform operations for updating language understanding classifier models, the operations comprising:
receiving from at least one computing device of a plurality of computing devices communicatively coupled to the server computer, a first user selection of at least one of the following: at least one intent of a plurality of available intents and/or at least one slot for the at least one intent, wherein:
the at least one intent is associated with at least one action used to perform at least one function of a category of functions for a domain;
the at least one slot indicating a value used for performing the at least one action; and
the first user selection associated with a digital voice input received at the at least one computing device; and

upon receiving from at least another computing device of the plurality of computing devices, a plurality of subsequent user selections that are identical to the first user selection and a plurality of subsequent digital voice inputs corresponding to the plurality of subsequent user selections, wherein the plurality of subsequent digital voice inputs are substantially similar to the digital voice input:
generating a labeled data set by pairing the digital voice input with the first user selection;
selecting a language understanding classifier from a plurality of available language understanding classifiers associated with one or more agent definitions, the selecting based at least on the at least one intent; and
updating the selected language understanding classifier based on the generated labeled data set.

2. The server computer according to claim 1 , the operations further comprising:
determining a number of the plurality of subsequent user selections; and
when the number of the plurality of subsequent user selections is higher than a first threshold, automatically updating the selected language understanding classifier based on the generated labeled data set.

3. The server computer according to claim 1 , the operations further comprising:
receiving the digital voice input from the at least one computing device;
performing natural language processing using the digital voice input to determine a user voice request; and
storing one or both of the digital voice input and the user voice request in an utterances database.

4. The server computer according to claim 3 , the operations further comprising:
retrieving one or both of the digital voice input and the user voice request from the utterances database;
generating the labeled data set by pairing the first user selection with one or both of the digital voice input and the user voice request; and
storing the generated labeled data set in a labeled data database.

5. The server computer according to claim 1 , the operations further comprising:
determining a number of the plurality of subsequent user selections which comprise at least one intent and at least one slot that are different from the at least one intent and the at least one slot of the first user selection.

6. The server computer according to claim 5 , the operations further comprising:
when the determined number of the plurality of subsequent user selections is higher than a second threshold, generating a request for manual updating of the selected language understanding classifier by an administrator of the server computer.

7. The server computer according to claim 6 , the operations further comprising:
in response to the request for manual updating, receiving input selecting the at least one intent and the at least one slot of the first user selection or the at least one intent and the at least one slot of the plurality of subsequent user selections.

8. The server computer according to claim 7 , the operations further comprising:
receiving input updating the selected language understanding classifier based on the selected at least one intent and the at least one slot.

9. A method for updating language understanding classifier models, the method comprising:
receiving via one or more microphones of a computing device, a digital voice input from a user of the computing device;
performing natural language processing using the digital voice input to determine a user voice request;
upon determining the user voice request does not match at least one of a plurality of pre-defined tasks in an agent definition of a digital personal assistant running on the computing device:
receiving using a graphical user interface of an end-user labeling tool (EULT) of the computing device, a user selection of at least one of the following: an intent of a plurality of available intents and at least one slot for the intent, wherein:
the intent is associated with at least one action used to perform at least one function of a category of functions for a domain; and
the at least one slot indicating a value used for performing the at least one action;

generating a labeled data set by pairing the user voice request and the user selection;
selecting a language understanding classifier from a plurality of available language understanding classifiers associated with the agent definition, the selecting based at least on the intent selected by the user; and
updating the selected language understanding classifier based on the generated labeled data set.

10. The method according to claim 9 , wherein the plurality of available language understanding classifiers associated with the agent definition are stored in local storage at the computing device and the method further comprises:
associating the updated language understanding classifier with a profile of the user within the computing device; and
storing the updated language understanding classifier in the local storage.

11. The method according to claim 10 , further comprising:
designating the updated language understanding classifier as a common resource that can be shared between the digital personal assistant and at least one third-party application running on the computing device.

12. The method according to claim 9 , wherein the updating comprises:
replacing an association of the selected language understanding classifier with at least one of a previous intent and/or slot with a new association with at least one of the intent and/or the at least one slot of the user selection.

13. The method according to claim 12 , further comprising:
creating an association of the selected language understanding classifier with one or both of the digital voice input and the user voice request.

14. The method according to claim 9 , wherein the user selection comprises the intent, and the method further comprises:
selecting the at least one slot automatically, based on the intent and the user voice request.

15. The method according to claim 9 , wherein the agent definition comprises at least one of a voice command definition (VCD) schema and a reactive agent definition (RAD) schema associated with the digital personal assistant.

16. A computer-readable storage medium storing computer-executable instructions for causing a computing device to perform operations for updating language understanding classifier models, the operations comprising:
determining a user request based on user input received at a computing device, the user request received via at least one of text input and voice input, the request for a functionality of a digital personal assistant running on the computing device;
determining the user request does not match at least o

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Kannan Vishwac Sena
• Uzelac Aleksandar
• Hwang Daniel J

Classifications

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G10L15/063 Primary

  Training · CPC title