Restructuring deep neural network acoustic models

Patent metadata

Field	Value
Publication number	US-9728184-B2
Application number	US-201313920323-A
Country	US
Kind code	B2
Filing date	Jun 18, 2013
Priority date	Jun 18, 2013
Publication date	Aug 8, 2017
Grant date	Aug 8, 2017

Abstract

A Deep Neural Network (DNN) model used in an Automatic Speech Recognition (ASR) system is restructured. A restructured DNN model may include fewer parameters compared to the original DNN model. The restructured DNN model may include a monophone state output layer in addition to the senone output layer of the original DNN model. Singular value decomposition (SVD) can be applied to one or more weight matrices of the DNN model to reduce the size of the DNN Model. The output layer of the DNN model may be restructured to include monophone states in addition to the senones (tied triphone states) which are included in the original DNN model. When the monophone states are included in the restructured DNN model, the posteriors of monophone states are used to select a small part of senones to be evaluated.

First claim

What is claimed is:

1. A method comprising:
accessing a Deep Neural Network (DNN) model that includes a weight matrix and layers comprising:
an input layer;
a first hidden layer;
a second hidden layer, wherein the first and second hidden layers are coupled by the weight matrix comprising a plurality of values; and
an output layer;

determining whether the weight matrix is a weight matrix having at least as many parameters as a weight matrix immediately preceding the output layer;
upon determining that the weight matrix has at least as many parameters as a weight matrix immediately preceding the output layer, reducing a sparseness of the weight matrix in the DNN model, wherein reducing the sparseness comprises executing decomposition processing of the weight matrix to generate two smaller matrices from the weight matrix, wherein the decomposition processing comprises applying Singular Value Decomposition (SVD) to the weight matrix;
restructuring the DNN model based on the executed decomposition processing, wherein the restructuring further comprises modifying the plurality of values coupling the first and second hidden layers of the DNN model by replacing the weight matrix with the two smaller matrices;
providing the restructured DNN model;
receiving an utterance; and
processing the received utterance using the restructured DNN model.

2. The method of claim 1 , wherein restructuring the DNN model with the weight matrix reduced in sparseness comprises splitting a layer in the DNN model into at least two smaller layers.

3. The method of claim 1 , wherein the restructuring further comprises replacing, in at least one layer of the DNN model, the weight matrix with the at least two smaller matrices.

4. The method of claim 1 , wherein the output layer comprises a senone output layer and a monophone state output layer.

5. The method of claim 1 , further comprising training the output layer of the DNN to use a monophone state.

6. The method of claim 1 , further comprising tuning the restructured model using a back-propagation method.

7. A computer storage device storing computer-executable instructions that, when executed by at least one processor, perform a method comprising:
creating a restructured Deep Neural Network (DNN) model from an original DNN model, wherein the creating further comprises:
accessing the original DNN model, the original DNN model including a weight matrix and layers comprising:
an input layer;
a first hidden layer;
a second hidden layer, wherein the first and second hidden layers are coupled by the weight matrix comprising a plurality of values; and
an output layer

determining whether the weight matrix is a weight matrix having at least as many parameters as a weight matrix immediately preceding the output layer;
upon determining that the weight matrix has at least as many parameters as a weight matrix immediately preceding the output layer, executing decomposition processing of the weight matrix of the original DNN model to generate two smaller matrices from the weight matrix, wherein the decomposition processing comprises applying Singular Value Decomposition (SVD) to the weight matrix; and
restructuring the original DNN model based on the executed decomposition processing, wherein the restructuring further comprises modifying the plurality of values coupling the first and second hidden layers of the DNN model by replacing the weight matrix with the two smaller matrices;
receiving an utterance; and
using the restructured DNN model to recognize the received utterance.

8. The computer storage device of claim 7 , wherein a sparseness of the weight matrix in the original DNN is reduced in the restructured DNN model.

9. The computer storage device of claim 7 , wherein the output layer of the restructured DNN comprises a monophone state output layer and a senone output layer.

10. The computer storage device of claim 9 , further comprising using posteriors of monophone states to select senones to be evaluated to reduce the number of calculations in the senone output layer.

11. The computer storage device of claim 7 , wherein the restructured DNN model comprises at least one additional layer as compared with the original DNN model.

12. The computer storage device of claim 7 , further comprising tuning the restructured DNN model by executing a back-propagation method before using the restructured DNN model.

13. A system comprising:
a processor and memory;
an operating environment executing using the processor; and
a model manager that is configured to perform actions comprising:
accessing a Deep Neural Network (DNN) model that includes a weight matrix and layers comprising:
an input layer;
a first hidden layer;
a second hidden layer, wherein the first and second hidden layers are coupled by the weight matrix comprising plurality of values; and
an output layer;

determining whether the weight matrix is a weight matrix having at least as many parameters as a weight matrix immediately preceding the output layer;
upon determining that the weight matrix has at least as many parameters as a weight matrix immediately preceding the output layer, reducing a sparseness of the weight matrix in the DNN model, wherein reducing the sparseness comprises executing decomposition processing of the weight matrix to generate two smaller matrices from the weight matrix, wherein the decomposition processing comprises applying Singular Value Decomposition (SVD) to the weight matrix;
restructuring the DNN model based on the executed decomposition processing, wherein the restructuring further comprises modifying the plurality of values coupling the first and second hidden layers of the DNN model by replacing the weight matrix with the two smaller matrices;
providing the restructured DNN model;
receiving an utterance; and
processing the received utterance using the restructured DNN model.

14. The system of claim 13 , wherein restructuring the DNN model with the weight matrix reduced in sparseness comprises splitting one of the layers in the DNN model into at least two smaller layers.

15. The system of claim 13 , wherein the output layer comprises a senone output layer and a monophone state output layer.

16. The system of claim 13 , further comprising training the output layer of the DNN to use a monophone state.

17. The method of claim 1 , wherein the weight matrix is automatically reduced if it is a weight matrix immediately preceding the output layer.

18. The system of claim 13 , wherein the weight matrix is automatically reduced if it is a weight matrix immediately preceding the output layer.

19. The computer storage device of claim 7 , wherein the instructions are further executable by the at least one processor for automatically reducing the weight matrix if it is a weight matrix immediately preceding the output layer.

20. The system of claim 13 , wherein the model manager is further configured to tune the restructured DNN model by executing a back-propagation method before using the restructured DNN model.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Xue Jian
• Stoimenov Emilian
• Li Jinyu
• Gong Yifan

Classifications

• G10L15/142

  Hidden Markov Models [HMMs] · CPC title

• G10L15/16 Primary

  using artificial neural networks · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title