Intent authoring using weak supervision and co-training for automated response systems

Patent metadata

Field	Value
Publication number	US-11568856-B2
Application number	US-202016949232-A
Country	US
Kind code	B2
Filing date	Oct 21, 2020
Priority date	Aug 31, 2018
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

A combination of propagation operations and learning algorithms is applied, using a selected set of labeled conversational logs retrieved from a subset of a plurality of conversational logs, to a remaining corpus of the plurality of conversational logs to train an automated response system according to an intent associated with each of the conversational logs. The combination of propagation operations and learning algorithms may include defining the labels by a user for the selected set of the subset of the plurality of conversational logs; training a probabilistic classifier using the defined labels of features of the selected set, wherein the probabilistic classifier produces labeling decisions for the subset of conversational logs; weighting the features of the selected set in a model optimization process; and/or training an additional classifier using the weighted features of the selected set and applying the additional classifier to the remaining corpus.

First claim

The invention claimed is:

1. A method for training an automated response system using weak supervision and co-training, by a processor, comprising:
applying a combination of propagation operations and learning algorithms, using a selected set of labeled conversational logs retrieved from a subset of a plurality of conversational logs, to a remaining corpus of the plurality of conversational logs to train the automated response system according to an intent associated with each of the conversational logs, wherein applying the combination of propagation operations and learning algorithms comprises:
defining the labels by a user for the selected set of the subset of the plurality of conversational logs;
training a probabilistic classifier using the defined labels of features of the selected set, wherein the probabilistic classifier produces labeling decisions for the subset of conversational logs;
weighting the features of the selected set in a model optimization process; and
training an additional classifier using the weighted features of the selected set and applying the additional classifier to the remaining corpus.

2. The method of claim 1 , wherein the subset of the plurality of conversational logs is returned in response to an input query by the user requesting to retrieve the subset of the plurality of conversational logs according to a defined criterion; and
the defined criterion comprises one or more utterances relevant to the intent input by the user during the input query.

3. The method of claim 2 , wherein the combination of propagation operations and learning algorithms are applied in parallel to the remaining corpus.

4. The method of claim 2 , wherein the combination of propagation operations and learning algorithms are applied sequentially to the remaining corpus such that an output of a first operation is used to train an input of a second operation performed on the remaining corpus.

5. The method of claim 2 , further including, in response to the input query by the user, presenting to the user suggested alternative queries to retrieve other utterances within the remaining corpus of the plurality of conversational logs relevant to the intent.

6. The method of claim 1 , wherein the labeling further includes displaying the selected set of the subset of conversational logs on a user interface (UI) and receiving user input indicating affirmatively or negatively whether each of the selected set of the subset of conversational logs is relevant to the intent.

7. The method of claim 1 , wherein the plurality of conversational logs are received from a historical repository of previously saved interactive dialog sessions.

8. A system for training automated response systems using weak supervision and co-training, comprising:
a processor executing instructions stored in a memory device; wherein the processor:
applies a combination of propagation operations and learning algorithms, using a selected set of labeled conversational logs retrieved from a subset of a plurality of conversational logs, to a remaining corpus of the plurality of conversational logs to train the automated response system according to an intent associated with each of the conversational logs, wherein applying the combination of propagation operations and learning algorithms comprises:
defining the labels by a user for the selected set of the subset of the plurality of conversational logs;
training a probabilistic classifier using the defined labels of features of the selected set, wherein the probabilistic classifier produces labeling decisions for the subset of conversational logs;
weighting the features of the selected set in a model optimization process; and
training an additional classifier using the weighted features of the selected set and applying the additional classifier to the remaining corpus.

9. The system of claim 8 , wherein the subset of the plurality of conversational logs is returned in response to an input query by the user requesting to retrieve the subset of the plurality of conversational logs according to a defined criterion; and
the defined criterion comprises one or more utterances relevant to the intent input by the user during the input query.

10. The system of claim 9 , wherein the combination of propagation operations and learning algorithms are applied in parallel to the remaining corpus.

11. The system of claim 9 , wherein the combination of propagation operations and learning algorithms are applied sequentially to the remaining corpus such that an output of a first operation is used to train an input of a second operation performed on the remaining corpus.

12. The system of claim 9 , wherein the processor, in response to the input query by the user, presents to the user suggested alternative queries to retrieve other utterances within the remaining corpus of the plurality of conversational logs relevant to the intent.

13. The system of claim 8 , wherein the labeling further includes displaying the selected set of the subset of conversational logs on a user interface (UI) and receiving user input indicating affirmatively or negatively whether each of the selected set of the subset of conversational logs is relevant to the intent.

14. The system of claim 8 , wherein the plurality of conversational logs are received from a historical repository of previously saved interactive dialog sessions.

15. A computer program product for training automated response systems using weak supervision and co-training, by a processor, the computer program product embodied on a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:
an executable portion that applies a combination of propagation operations and learning algorithms, using a selected set of labeled conversational logs retrieved from a subset of a plurality of conversational logs, to a remaining corpus of the plurality of conversational logs to train the automated response system according to an intent associated with each of the conversational logs, wherein applying the combination of propagation operations and learning algorithms comprises:
defining the labels by a user for the selected set of the subset of the plurality of conversational logs;
training a probabilistic classifier using the defined labels of features of the selected set, wherein the probabilistic classifier produces labeling decisions for the subset of conversational logs;
weighting the features of the selected set in a model optimization process; and
training an additional classifier using the weighted features of the selected set and applying the additional classifier to the remaining corpus.

16. The computer program product of claim 15 , wherein the subset of the plurality of conversational logs is returned in response to an input query by the user requesting to retrieve the subset of the plurality of conversational logs according to a defined criterion; and
the defined criterion comprises one or more utterances relevant to the intent input by the user during the input query.

17. The computer program product of claim 16 , wherein the combination of propagation operations and learning algorithms are applied in parallel to the remaining corpus.

18. The computer program product of claim 16 , wherein the combination of propagation operations and learning algorithms are applied sequentially to the remaining corpus such that an output of a first operation is used to train an input of a second operation performed on the rem

Assignees

• IBM

Inventors

• Ho Tin Kam
• Yates Robert L
• Mcgregor Blake
• Ugrani Rajendra G
• Mallinar Neil R
• Shah Abhishek
• Gupta Ayush

Classifications

• G10L15/1815

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G10L2015/0638

  Interactive procedures · CPC title

• G10L15/063 Primary

  Training · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title