Intent authoring using weak supervision and co-training for automated response systems

Patent metadata

Field	Value
Publication number	US-10832659-B2
Application number	US-201816119849-A
Country	US
Kind code	B2
Filing date	Aug 31, 2018
Priority date	Aug 31, 2018
Publication date	Nov 10, 2020
Grant date	Nov 10, 2020

Abstract

Embodiments for training an automated response system using weak supervision and co-training in a computing environment are provided. A plurality of conversational logs comprising interactive dialog sessions between agents and clients for a given product or service are received. A subset of the plurality of conversational logs are retrieved according to a defined criterion, and a selected set of the subset of the plurality of retrieved conversational logs are labeled by a user. The labeling is associated with a semantic scope of intent considered by the clients. A combination of propagation operations and learning algorithms using the selected set of labeled conversational logs are applied to a remaining corpus of the plurality of conversational logs to train the automated response system according to the semantic scope of intent.

First claim

The invention claimed is:

1. A method for training an automated response system using weak supervision and co-training, by a processor, comprising:
receiving a plurality of conversational logs comprising interactive dialog sessions between agents and clients for a given product or service;
retrieving a subset of the plurality of conversational logs according to a defined criterion, wherein the subset of the plurality of conversational logs is retrieved in response to an input query by a user such that the defined criterion comprises one or more utterances relevant to the semantic scope of intent input by the user during the input query;
labeling, by the user, a selected set of the subset of the plurality of retrieved conversational logs, the labeling associated with a semantic scope of intent considered by the clients; and
applying a combination of propagation operations and learning algorithms using the selected set of labeled conversational logs to a remaining corpus of the plurality of conversational logs to train the automated response system according to the semantic scope of intent, wherein applying the combination of propagation operations and learning algorithms further comprises:
defining the labels by the user for the selected set of the subset of conversational logs comprising retrieved utterances to the subset of conversational logs according to the input query;
training a probabilistic classifier using the defined labels of features of the retrieved utterances; wherein the probabilistic classifier produces labeling decisions for the subset of conversational logs;
weighting the features of the retrieved utterances in a model optimization process; and
training an additional classifier using the weighted features of the retrieved utterances and applying the additional classifier to the remaining corpus.

2. The method of claim 1 , wherein the labeling further includes displaying the selected set of the subset of conversational logs on a user interface (UI) and receiving user input indicating affirmatively or negatively whether each of the selected set of the subset of conversational logs is relevant to the semantic scope of intent.

3. The method of claim 1 , wherein the combination of propagation operations and learning algorithms are applied in parallel to the remaining corpus; or
the combination of propagation operations and learning algorithms are applied sequentially to the remaining corpus such that an output of a first operation is used to train an input of a second operation performed on the remaining corpus.

4. The method of claim 1 , wherein the plurality of conversational logs are received from a historical repository of previously saved interactive dialog sessions; and
the agents comprise at least one of human operators and virtual operators.

5. The method of claim 1 , further including, in response to the input query by the user, presenting to the user suggested alternative queries to retrieve other utterances within the remaining corpus of the plurality of conversational logs relevant to the semantic scope of intent.

6. A system for training automated response systems using weak supervision and co-training, comprising:
a processor executing instructions stored in a memory device; wherein the processor:
receives a plurality of conversational logs comprising interactive dialog sessions between agents and clients for a given product or service;
retrieves a subset of the plurality of conversational logs according to a defined criterion, wherein the subset of the plurality of conversational logs is retrieved in response to an input query by a user such that the defined criterion comprises one or more utterances relevant to the semantic scope of intent input by the user during the input query;
receives input of the user labeling a selected set of the subset of the plurality of retrieved conversational logs, the labeling associated with a semantic scope of intent considered by the clients; and
applies a combination of propagation operations and learning algorithms using the selected set of labeled conversational logs to a remaining corpus of the plurality of conversational logs to train the automated response system according to the semantic scope of intent, wherein applying the combination of propagation operations and learning algorithms further comprises:
defining the labels by the user for the selected set of the subset of conversational logs comprising retrieved utterances to the subset of conversational logs according to the input query;
training a probabilistic classifier using the defined labels of features of the retrieved utterances; wherein the probabilistic classifier produces labeling decisions for the subset of conversational logs;
weighting the features of the retrieved utterances in a model optimization process; and
training an additional classifier using the weighted features of the retrieved utterances and applying the additional classifier to the remaining corpus.

7. The system of claim 6 , wherein the labeling further includes displaying the selected set of the subset of conversational logs on a user interface (UI) and receiving user input indicating affirmatively or negatively whether each of the selected set of the subset of conversational logs is relevant to the semantic scope of intent.

8. The system of claim 6 , wherein the combination of propagation operations and learning algorithms are applied in parallel to the remaining corpus; or
the combination of propagation operations and learning algorithms are applied sequentially to the remaining corpus such that an output of a first operation is used to train an input of a second operation performed on the remaining corpus.

9. The system of claim 6 , wherein the plurality of conversational logs are received from a historical repository of previously saved interactive dialog sessions; and
the agents comprise at least one of human operators and virtual operators.

10. The system of claim 6 , wherein the processor, in response to the input query by the user, presents to the user suggested alternative queries to retrieve other utterances within the remaining corpus of the plurality of conversational logs relevant to the semantic scope of intent.

11. A computer program product for training automated response systems using weak supervision and co-training, by a processor, the computer program product embodied on a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:
an executable portion that receives a plurality of conversational logs comprising interactive dialog sessions between agents and clients for a given product or service;
an executable portion that retrieves a subset of the plurality of conversational logs according to a defined criterion, wherein the subset of the plurality of conversational logs is retrieved in response to an input query by a user such that the defined criterion comprises one or more utterances relevant to the semantic scope of intent input by the user during the input query;
an executable portion that receives input of the user labeling a selected set of the subset of the plurality of retrieved conversational logs, the labeling associated with a semantic scope of intent considered by the clients; and
an executable portion that applies a combination of propagation operations and learning algorithms using the selected set of labeled conversational logs to a remaining corpus of the plurality of conversational logs to train the automated response system according to the semantic scope of intent, wherein applying the combination of propagation operations and

Assignees

• IBM

Inventors

• Ho Tin Kam
• Yates Robert L
• Mcgregor Blake
• Ugrani Rajendra G
• Mallinar Neil R
• Shah Abhishek
• Gupta Ayush

Classifications

• G06F40/30 Primary

  Semantic analysis · CPC title

• G10L15/063 Primary

  Training · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/1815

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title