Deep neural network-based decision network

Patent metadata

Field	Value
Publication number	US-11250311-B2
Application number	US-201715853570-A
Country	US
Kind code	B2
Filing date	Dec 22, 2017
Priority date	Mar 15, 2017
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

The technology disclosed proposes using a combination of computationally cheap, less-accurate bag of words (BoW) model and computationally expensive, more-accurate long short-term memory (LSTM) model to perform natural processing tasks such as sentiment analysis. The use of cheap, less-accurate BoW model is referred to herein as “skimming”. The use of expensive, more-accurate LSTM model is referred to herein as “reading”. The technology disclosed presents a probability-based guider (PBG). PBG combines the use of BoW model and the LSTM model. PBG uses a probability thresholding strategy to determine, based on the results of the BoW model, whether to invoke the LSTM model for reliably classifying a sentence as positive or negative. The technology disclosed also presents a deep neural network-based decision network (DDN) that is trained to learn the relationship between the BoW model and the LSTM model and to invoke only one of the two models.

First claim

What is claimed is:

1. A system with numerous processors operating in parallel, comprising:
a memory storing a plurality of processor-executable instructions for training and operating a decision neural network-based classifier; and
a processor that reads the plurality of processor-executable instructions from the memory to:
select, by the decision neural network-based classifier, on an input-by-input basis, between a non-recurrent neural network-based classifier and a recurrent neural network-based classifier to perform a machine classification task, and
train the decision neural network-based classifier to perform the selection using a decision training set annotated with model class labels, wherein the model class labels distinguish between inputs accurately classified only by a trained recurrent neural network-based classifier and remaining inputs from the decision training set,

wherein the decision training set is generated by:
identifying, based on a confusion matrix generated from performance of the non-recurrent and recurrent neural network-based classifiers, a first subset of validation inputs inferred only by the recurrent neural network-based classifier and a second subset of validation inputs comprising inputs not in the first subset, and
including, in the decision training set, the first subset of validation inputs labeled with a first model class label identifying the recurrent neural network-based classifier and the second subset of validation inputs labeled with a second model class label identifying the non-recurrent neural network-based classifier.

2. The system of claim 1 , wherein the remaining inputs include inputs inaccurately classified by the trained recurrent neural network-based classifier and inputs accurately classified by both the trained non-recurrent and recurrent neural network-based classifiers.

3. The system of claim 1 , wherein the processor further reads the plurality of processor-executable instructions from the memory to:
train the non-recurrent and recurrent neural network-based classifiers to perform the machine classification task using a training set, the training set comprising training inputs annotated with task class labels defined for the machine classification task;
use the trained non-recurrent and recurrent neural network-based classifiers to perform the machine classification task on a validation set, the validation set comprising validation inputs annotated with the task class labels; and
train the decision neural network-based classifier using the decision training set to output probabilities for the first and second model class labels on an input-by-input basis, the output probabilities specifying respective likelihoods of selecting the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier.

4. The system of claim 1 , wherein the processor further reads the plurality of processor-executable instructions from the memory to:
for a given input, during inference, based on output probabilities of the trained decision neural network-based classifier, perform the machine classification task on the given input using either the trained recurrent neural network-based classifier or the trained non-recurrent neural network-based classifier.

5. The system of claim 3 , wherein the processor further reads the plurality of processor-executable instructions from the memory to select the trained recurrent neural network-based classifier when the output probability of the first model class label is higher than that of the second model class label.

6. The system of claim 3 , wherein the processor further reads the plurality of processor-executable instructions from the memory to select the trained non-recurrent neural network-based classifier when the output probability of the second model class label is higher than that of the first model class label.

7. The system of claim 1 , wherein the trained recurrent neural network-based classifier is at least three percent more accurate and four times computationally more expensive than the trained non-recurrent neural network-based classifier.

8. The system of claim 7 , wherein the trained recurrent neural network-based classifier is at least one recurrent neural network (abbreviated RNN).

9. The system of claim 7 , wherein the trained non-recurrent neural network-based classifier is at least one bag of words (abbreviated BoW) network.

10. The system of claim 7 , wherein the trained non-recurrent neural network-based classifier is at least one continuous bag of words (abbreviated CBoW) network.

11. The system of claim 7 , wherein the trained non-recurrent neural network-based classifier is at least one skip-gram network.

12. The system of claim 7 , wherein the trained non-recurrent neural network-based classifier is at least one convolutional neural network (abbreviated CNN).

13. The system of claim 8 , wherein the RNN is a long short-term memory (abbreviated LSTM) network.

14. The system of claim 8 , wherein the RNN is a gated recurrent unit (abbreviated GRU) network.

15. The system of claim 8 , wherein the RNN is a quasi-recurrent neural network (abbreviated QRNN).

16. The system of claim 3 , wherein the training set and the validation set are part of a single data set that is subjected to held-out splitting to create the training set and the validation set.

17. The system of claim 3 , wherein the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier are trained separately on the training set.

18. The system of claim 1 , wherein the machine classification task is sentiment classification and the inputs are sentences.

19. The system of claim 18 , wherein the task class labels are at least one of positive sentiment, negative sentiment, very positive sentiment, very negative sentiment, somewhat positive sentiment, somewhat negative sentiment, or neutral sentiment.

20. The system of claim 3 , wherein the confusion matrix identifies at least one of:
validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier;
validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier;
validation inputs accurately classified by the trained non-recurrent neural network-based classifier but inaccurately classified by the trained recurrent neural network-based classifier; and
validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier.

21. The system of claim 3 , wherein the first subset includes validation inputs accurately classified by the trained recurrent neural network-based classifier but inaccurately classified by the trained non-recurrent neural network-based classifier.

22. The system of claim 3 , wherein the second subset includes at least one of:
validation inputs accurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier;
validation inputs inaccurately classified by both the trained recurrent neural network-based classifier and the trained non-recurrent neural network-based classifier; and
validation inputs accurately classified by the trained non-recurrent neural netw

Assignees

• Salesforce Com Inc

Inventors

• Johansen Alexander Rosenberg
• Mccann Bryan
• Bradbury James
• Socher Richard

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/24133

  Distances to prototypes · CPC title