Generating computer responses to social conversational inputs

Patent metadata

Field	Value
Publication number	US-9547471-B2
Application number	US-201414323050-A
Country	US
Kind code	B2
Filing date	Jul 3, 2014
Priority date	Jul 3, 2014
Publication date	Jan 17, 2017
Grant date	Jan 17, 2017

Abstract

Conversational interactions between humans and computer systems can be provided by a computer system that classifies an input by conversation type, and provides human authored responses for conversation types. The input classification can be performed using trained binary classifiers. Training can be performed by labeling inputs as either positive or negative examples of a conversation type. Conversational responses can be authored by the same individuals that label the inputs used in training the classifiers. In some cases, the process of training classifiers can result in a suggestion of a new conversation type, for which human authors can label inputs for a new classifier and write content for responses for that new conversation type.

First claim

What is claimed is:

1. A process for training a computer-implemented classifier to classify conversational inputs into a conversation type, wherein the classifier has an input receiving data representing a conversational input and an output providing a match output indicating how the conversational input matches the conversation type represented by the classifier, the process for training the classifier being performed by a processing system comprising a processor and computer storage, the process comprising:
receiving, into the computer storage, data representing a first set of conversational inputs and including first label data indicating the first set as positive examples of the conversation type, and data representing a second set of conversational inputs and including second label data indicating the second set as negative examples of the conversation type;
building the classifier using the data representing the first and second sets of conversational inputs and the first label data and the second label data;
accessing, from the computer storage, a data representing a third set of conversational inputs;
for each conversational input in the third set of conversational inputs:
applying the data representing the conversational input to the inputs of the classifier to obtain a match output from the output of the classifier, and
in response to a determination that the match output indicates no classification of the conversational input as one of a negative example or a positive example:
presenting the conversational input through one or more user computers to one or more individuals for labeling,
receiving third label data through the one or more user computers from the one or more individuals for the presented conversational input, and
storing the received third label data in association with the conversation input in the computer storage; and

retraining the classifier using the data representing the third set of conversational inputs having the third label data.

2. The process of claim 1 , further comprising:
receiving and storing in the computer storage, from one or more of the one or more individuals, data representing conversational responses for the conversation type associated with the classifier for which the one or more individuals performed the labeling.

3. The process of claim 1 , further comprising:
identifying additional conversation types from the corpus data representing the third set of conversational inputs.

4. The process of claim 3 , wherein identifying comprises:
clustering conversational inputs in the data representing the third set of conversational inputs;
assigning a different conversation type to each cluster.

5. The process of claim 3 , wherein identifying comprises:
applying the conversational inputs to a plurality of classifiers;
identifying the conversational inputs for which none of the plurality of classifiers indicates a match;
assigning one or more new conversation types to the identified conversational inputs.

6. The process of claim 1 , wherein the conversation type of a classifier is included in a hierarchy of conversation types.

7. The process of claim 1 , further comprising automatically clustering the data representing the third set of conversational inputs to provide class information for the conversational inputs.

8. The process of claim 1 , wherein building the classifier comprises:
defining, with the processing system, a set of parameters for a metric applied by the classifiers to data applied to the inputs of the classifier.

9. The process of claim 8 , wherein the metric applied by the classifier comprises computer program instructions processed by a processor to compute a similarity metric or distance metric or probability metric using the data applied to the inputs to the classifier according to the set of parameters.

10. The process of claim 1 , wherein the data representing a conversational input in the first, second and third sets of conversation inputs comprises a plurality of features derived from the conversational input.

11. A computer system comprising:
a processing system comprising a processor and computer storage;
a classifier to classify conversational inputs into a conversation type, wherein the classifier has an input receiving data representing a conversational input and an output providing a match output indicating how the conversational input matches the conversation type represented by the classifier;
the computer storage storing data representing a first set of conversational inputs and including first label data indicating the first set as positive examples of the conversation type, and data representing a second set of conversational inputs and including second label data indicating the second set as negative examples of the conversation type;
a training module to build the classifier using the data representing the first and second sets of conversational inputs and the first label data and the second label data and configured to:
access, from the computer storage, data representing a third set of conversational inputs;
for each conversational input in the third set of conversational inputs:
apply the data representing the conversational input to the inputs of the classifier to obtain a match output from the output of the classifier, and
in response to a determination that the match output indicates no classification of the conversational input as one of a negative example or a positive example:
present the conversational input through one or more user computers to one or more individuals for labeling,
receive third label data through the one or more user computers from the one or more individuals for the presented conversational input, and
store the received third label data in association with the conversation input in the computer storage; and

retrain the classifier using the data representing the third set of conversational inputs having the third label data.

12. The computer system of claim 11 , wherein the training module is further configured, for building the classifier, to define a set of parameters for a metric applied by the classifiers to data applied to the inputs of the classifier.

13. The computer system of claim 12 , wherein the metric applied by the classifier comprises computer program instructions processed by a processor to compute a similarity metric or distance metric or probability metric using the data applied to the inputs to the classifier according to the set of parameters.

14. The computer system of claim 11 , wherein the data representing a conversational input in the first, second and third sets of conversation inputs comprises a plurality of features derived from the conversational input.

15. The computer system of claim 11 , wherein the training module is further configured to receive and store in the computer storage, from one or more of the one or more individuals, data representing conversational responses for the conversation type associated with the classifier for which the one or more individuals performed the labeling.

16. The computer system of claim 12 , wherein the training module is further configured to identify an additional conversation type based on processing the data representing the third set of conversational inputs.

17. The computer system of claim 16 , to identify an additional conversation type, the training module is further configured to:
cluster conversational inputs in the third set of conversational inputs into clusters based on the data representing the third set of conversational inputs; and
assign a conve

Assignees

• Microsoft Corp
• Microsoft Technology Licensing Llc

Inventors

• Williams Jason
• Zweig Geoffrey
• Lakshmiratan Aparna
• Suarez Carlos Garcia Jurado

Classifications

• G10L15/1822 Primary

  Parsing for meaning understanding · CPC title

• G06F40/35

  Discourse or dialogue representation · CPC title

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L2015/226

  using non-speech characteristics · CPC title