Query rejection for language understanding

Patent metadata

Field	Value
Publication number	US-2018349794-A1
Application number	US-201715611104-A
Country	US
Kind code	A1
Filing date	Jun 1, 2017
Priority date	Jun 1, 2017
Publication date	Dec 6, 2018
Grant date	—

Abstract

Techniques are provided for rejecting out-of-domain (OD) queries in a language understanding system. A methodology implementing the techniques according to an embodiment includes generating a plurality of in-domain (ID) utterances based on variations of provided ID sentences, and generating a plurality of OD utterances based on variations of provided OD sentences. The method may further include training an ID language model based on the generated ID utterances and training an OD language model based on the generated OD utterances. The ID language model is configured to generate an ID dataset based on calculated probabilities associated with the generated ID utterances. The OD language model is configured to generate an OD dataset based on calculated probabilities associated with the generated OD utterances. The method further includes training a classifier to detect OD queries from a plurality of received queries, the training based on the ID dataset and the OD dataset.

First claim

What is claimed is:

1 . At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for training a classifier to detect out-of-domain queries, the operations comprising:
generating a plurality of in-domain (ID) utterances based on variations of one or more of a plurality of ID sentences; generating a plurality of out-of-domain (OD) utterances based on variations of one or more of a plurality of OD sentences; generating an ID dataset based on calculated probabilities associated with the plurality of ID utterances; generating an OD dataset based on calculated probabilities associated with the plurality of OD utterances; training a classifier to detect OD queries from a received plurality of queries, the training based on the ID dataset and the OD dataset; and rejecting one or more of the detected OD queries.

2 . The computer readable storage medium of claim 1 , wherein the classifier detection further includes a probability estimate associated with the detected OD query.

3 . The computer readable storage medium of claim 1 , the operations further comprising rejecting one or more of the detected OD queries and providing one or more non-rejected queries to a language-based application.

4 . The computer readable storage medium of claim 1 , the operations further comprising:
generating the variations of the ID sentences by substituting one or more words selected from the ID sentences with synonyms associated with the selected words from the ID sentences; and generating the variations of the OD sentences by substituting one or more words selected from the OD sentences with synonyms associated with the selected words from the OD sentences.

5 . The computer readable storage medium of claim 1 , the operations further comprising generating the variations by inserting a value into the ID sentences or the OD sentences, the value associated with properties of words of the ID sentences or the OD sentences, the value selected from a pre-defined range of values.

6 . The computer readable storage medium of claim 1 , the operations further comprising generating the variations by inserting a phrase into the ID sentences or the OD sentences, the phrase generated based on parts-of-speech rules and probabilistic rules.

7 . The computer readable storage medium of claim 1 , the operations further comprising:
recognizing a class relationship between a first phrase in a first sentence, of the ID sentences or the OD sentences, and a second phrase in a second sentence, of the ID sentences or the OD sentences, the recognition based on predetermined rules; and generating the variations based on the class relationship.

8 . The computer readable storage medium of claim 1 , the operations further comprising:
generating feature vectors for words of the ID sentences and/or words of the OD sentences; performing dimension reduction of the feature vectors, the dimension reduction based on at least one of application of a neural network, principal component analysis, and linear discriminant analysis; recognizing a class relationship between a first of the words and a second of the words, the recognition based on the dimension reduced feature vectors; and generating the variations based on the class relationship.

9 . The computer readable storage medium of claim 1 , wherein at least one of:
generating an ID dataset includes the operation of training an ID language model based on the plurality of ID utterances, the ID language model to generate the ID dataset based on calculated probabilities associated with the plurality of ID utterances; generating an OD dataset includes the operation of training an OD language model based on the plurality of OD utterances, the OD language model to generate an OD dataset based on calculated probabilities associated with the plurality of OD utterances; and the ID language model and the OD language model are implemented as at least one of a recurrent neural network or a Markov N-gram model.

10 . The computer readable storage medium of claim 9 , wherein the training of the ID language model is based on at least one of words, letters, and phoneme sequences derived from the plurality of ID utterances; and the training of the OD language model is based on at least one of words, letters, and phoneme sequences derived from the plurality of OD utterances.

11 . The computer readable storage medium of claim 1 , wherein the classifier detection is further based on at least one of an automatic speech recognition (ASR) confidence indicator, a language model score, and an acoustic model score.

12 . The computer readable storage medium of claim 1 , further comprising the operations of receiving user feedback associated with the classifier detection of previous user queries, and iteratively adapting the training of the classifier based on the feedback.

13 . A system for training a classifier to detect out-of-domain queries, the system comprising:
an in-domain (ID) utterance generation circuit to generate a plurality of ID utterances based on variations of one or more of a plurality of ID sentences; an out-of-domain (OD) utterance generation circuit to generate a plurality of OD utterances based on variations of one or more of a plurality of OD sentences; an ID language model circuit to generate an ID dataset based on calculated probabilities associated with the plurality of ID utterances; an OD language model circuit to generate an OD dataset based on calculated probabilities associated with the plurality of OD utterances; and a classifier training circuit to train a classifier to detect OD queries from a received plurality of queries, the training based on the ID dataset and the OD dataset.

14 . The system of claim 13 , wherein the classifier is further to generate a probability estimate associated with the detected OD query.

15 . The system of claim 13 , wherein the classifier is further to reject one or more of the detected OD queries and provide one or more non-rejected queries to a language-based application.

16 . The system of claim 13 , further comprising an extrinsic generalization circuit to:
generate the variations of the ID sentences by substituting one or more words selected from the ID sentences with synonyms associated with the selected words from the ID sentences; and generate the variations of the OD sentences by substituting one or more words selected from the OD sentences with synonyms associated with the selected words from the OD sentences.

17 . The system of claim 13 , further comprising an extrinsic generalization circuit to generate the variations by inserting a value into the ID sentences or the OD sentences, the value associated with properties of words of the ID sentences or the OD sentences, the value selected from a pre-defined range of values.

18 . The system of claim 13 , further comprising an extrinsic generalization circuit to generate the variations by inserting a phrase into the ID sentences or the OD sentences, the phrase generated based on parts-of-speech rules and probabilistic rules.

19 . The system of claim 13 , further comprising an intrinsic generalization circuit to:
recognize a class relationship between a first phrase in a first sentence, of the ID sentences or the OD sentences, and a second phrase in a second sentence, of the ID sentences or the OD sentences, the recognition based on predetermined rules; and generate the variations

Assignees

• Intel Ip Corp

Inventors

• Georges Munir Nikolai Alexander
• Jessa Szymon
• Stemmer Georg

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06F40/30

  Semantic analysis · CPC title