External word embedding neural network language models

Patent metadata

Field	Value
Publication number	US-10019438-B2
Application number	US-201615074354-A
Country	US
Kind code	B2
Filing date	Mar 18, 2016
Priority date	Mar 18, 2016
Publication date	Jul 10, 2018
Grant date	Jul 10, 2018

Abstract

A mechanism is provided in a data processing system for external word embedding neural network language models. The mechanism configures the data processing system with an external word embedding neural network language model that accepts as input a sequence of words and predicts a current word based on the sequence of words. The external word embedding neural network language model combines an external embedding matrix to a history word embedding matrix and a prediction word embedding matrix of the external word embedding neural network language model. The mechanism receives a sequence of input words by the data processing system. The mechanism applies a plurality of previous words in the sequence of input words as inputs to the external word embedding neural network language model. The external word embedding neural network language model generates a predicted current word based on the plurality of previous words. The mechanism processes a current word in the sequence of input words based on the predicted current word generated by the external word embedding neural network language model.

First claim

What is claimed is:

1. A method, in a data processing system comprising a processor and a memory, the memory comprising instructions that are executed by the processor to specifically configure the processor to implement an automatic speech recognition system comprising an acoustic model and an external word embedding neural network language model, the method comprising:
configuring the automatic speech recognition system with an external word embedding neural network language model that accepts as input a sequence of words and predicts a current word based on the sequence of words, wherein the external word embedding neural network language model combines an external embedding matrix to a history word embedding matrix and a prediction word embedding matrix of the external word embedding neural network language model;
receiving as input an audio signal and generating a sequence of words based on the audio signal by the acoustic model;
receiving the sequence of input words by the automatic speech recognition system from the acoustic model;
applying a plurality of previous words in the sequence of input words as inputs to the external word embedding neural network language model in the automatic speech recognition system, wherein the external word embedding neural network language model generates a predicted current word based on the plurality of previous words; and
processing, by the automatic speech recognition system, a current word in the sequence of input words based on the predicted current word generated by the external word embedding neural network language model, wherein processing the current word comprises recognizing a current spoken word in the audio signal based on the predicted current word.

2. The method of claim 1 , wherein the external word embedding neural network language model generates input features from the plurality of previous words based on a concatenation of the external embedding matrix to the history word embedding matrix and provides the input features to a hidden layer of the external word embedding neural network language model.

3. The method of claim 2 , wherein the plurality of previous words are provided to the external word embedding neural network as word history vectors, w i−2 and w i−1 , and wherein the external embedding neural network language model generates the input features, x i , as follows:
x i =[w i−2 R H G w i−1 R H G ],

where R H G is the concatenation of the external embedding matrix to the history word embedding matrix.

4. The method of claim 3 , wherein the external embedding neural network language model determines the concatenation of the external embedding matrix to the history word embedding matrix, R H G , as follows:
R H G =[G R H ],

where G is the external word embedding matrix and R H is the history word embedding matrix.

5. The method of claim 2 , wherein a hidden layer of the external word embedding neural network language model applies a hidden layer weight matrix to the input features.

6. The method of claim 5 , wherein the external embedding neural network language model determines an output of the hidden layer, h i , as follows:
h i =σ( x i W ),

where σ is a point-wise sigmoid function, x i represents the input features, and W is the hidden layer weight matrix.

7. The method of claim 5 , wherein the external word embedding neural network language model generates the predicted current word using weight expansion of the hidden layer based on a concatenation of the external embedding matrix to the prediction word embedding matrix.

8. The method of claim 7 , wherein the external embedding neural network language model determines the predicted current word as a conditional probability mass function as follows:
P ( w i |w i−2 , w i−1 )= s ( h i R P G ),

where s is a soft-max function, h i is an output of the hidden layer, and R P G is the concatenation of the external embedding matrix to the prediction word embedding matrix.

9. The method of claim 8 , wherein the external embedding neural network language model determines the concatenation of the external embedding matrix to the prediction word embedding matrix, R P G as follows:
R P G =[G R P T ] T ,

where G is the external word embedding matrix and R P is the prediction word embedding matrix.

10. The method of claim 1 , wherein the external embedding matrix is generated using a semantic word embedding algorithm.

11. The method of claim 1 , wherein the external embedding neural language model is a 3-gram feed-forward neural network language model.

12. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program comprises:
an acoustic model that receives as input an audio signal and generates a sequence of words based on the audio signal by the acoustic model;
an external word embedding neural network language model that accepts as input a sequence of words and predicts a current word based on the sequence of words, wherein the external word embedding neural network language model combines an external embedding matrix to a history word embedding matrix and a prediction word embedding matrix of the external word embedding neural network language model;
instructions configured to execute on a computing device to cause the computing device to implement an automatic speech recognition system, wherein the instructions cause the computing device to:
receive the sequence of input words by the automatic speech recognition system from the acoustic model;
apply a plurality of previous words in the sequence of input words as inputs to the external word embedding neural network language model by the automatic speech recognition system, wherein the external word embedding neural network language model generates a predicted current word based on the plurality of previous words; and
process a current word in the sequence of input words based on the predicted current word generated by the external word embedding neural network language model in the automatic speech recognition system, wherein processing the current word comprises recognizing a current spoken word in the audio signal based on the predicted current word.

13. The computer program product of claim 12 , wherein the external word embedding neural network language model generates input features from the plurality of previous words based on a concatenation of the external embedding matrix to the history word embedding matrix and provides the input features to a hidden layer of the external word embedding neural network language model.

14. The computer program product of claim 13 , wherein a hidden layer of the external word embedding neural network language model applies a hidden layer weight matrix to the input features.

15. The computer program product of claim 14 , wherein the external word embedding neural network language model generates the predicted current word using weight expansion of the hidden layer based on a concatenation of the external embedding matrix to the prediction word embedding matrix.

16. The computer program product of claim 12 , wherein the external embedding matrix is generated using a semantic word embedding algorithm.

17. The computer program product of claim 12 , wherein the external embedding neural language model is a 3-gram feed-forward neural network language model.

18. An apparatus comprising:
a processor; and
a memory coupled to the proces

Assignees

• IBM

Inventors

• Audhkhasi Kartik
• Ramabhadran Bhuvana
• Sethy Abhinav

Classifications

• G06F40/30

  Semantic analysis · CPC title

• G06F40/289 Primary

  Phrasal analysis, e.g. finite state techniques or chunking · CPC title

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F40/253

  Grammatical analysis; Style critique · CPC title

• G10L15/16

  using artificial neural networks · CPC title