Methods and apparatus for training an artificial neural network for use in speech recognition

Patent metadata

Field	Value
Publication number	US-9627532-B2
Application number	US-201414308054-A
Country	US
Kind code	B2
Filing date	Jun 18, 2014
Priority date	Jun 18, 2014
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

Methods and apparatus for training a multi-layer artificial neural network for use in speech recognition. The method comprises determining for a first speech pattern of the plurality of speech patterns, using a first processing pipeline, network activations for a plurality of nodes of the artificial neural network in response to providing the first speech pattern as input to the artificial neural network, determining based, at least in part, on the network activations and a selection criterion, whether the artificial neural network should be trained on the first speech pattern, and updating, using a second processing pipeline, network weights between nodes of the artificial neural network based, at least in part, on the network activations when it is determined that the artificial neural network should be trained on the first speech pattern.

First claim

What is claimed is:

1. An apparatus configured to train a multi-layer artificial neural network for use in speech recognition, wherein each of the layers in the artificial neural network includes a plurality of nodes, each of the plurality of nodes in a layer being connected to a plurality of nodes in one or more adjacent layers of the artificial neural network, wherein the connections between nodes in the artificial neural network are associated with network weights, the apparatus comprising:
at least one processor programmed to implement a first processing pipeline and a second processing pipeline for training the artificial neural network based, at least in part, on a plurality of speech patterns provided as input to the artificial neural network;
wherein the first processing pipeline is configured to determine for a first speech pattern of the plurality of speech patterns provided as input to the artificial neural network, network activations for the plurality of nodes of the artificial neural network;
wherein the at least one processor is further programmed to determine based, at least in part, on the network activations and a selection criterion, whether the artificial neural network should be trained on the first speech pattern, wherein determining whether the artificial neural network should be trained on the first speech pattern comprises:
determining an output error for the first speech pattern based, at least in part, on the network activations; and
determining that the artificial neural network should be trained on the first speech pattern when the output error is greater than a threshold value; and

wherein the second processing pipeline is configured to update the network weights based, at least in part, on the network activations when it is determined that the artificial neural network should be trained on the first speech pattern.

2. The apparatus of claim 1 , wherein the selection criterion comprises a focused attention backpropagation criterion, and wherein determining whether the artificial neural network should be trained on the first speech pattern comprises applying the focused attention back propagation criterion to the network activations.

3. The apparatus of claim 1 , wherein the at least one processor is further programmed to:
determine based, at least in part, on the selection criterion, whether the first speech pattern has already been learned by the artificial neural network; and
copy data corresponding to the first speech pattern from the first processing pipeline to the second processing pipeline when it is determined that the first speech pattern has not already been learned by the artificial neural network.

4. The apparatus of claim 1 , wherein determining the output error comprises determining a mean squared error based, at least in part, on the network activations.

5. The apparatus of claim 3 , wherein copying data corresponding to the first speech pattern comprises determining whether a processing buffer of the second processing pipeline is full and copying the data when it is determined that the processing buffer is not full.

6. The apparatus of claim 5 , wherein the at least one processor is further programmed to update the network weights in response to determining that the processing buffer is full.

7. The apparatus of claim 3 , wherein copying the data comprises creating a pointer in a data structure of the second processing pipeline that references a location of the data in a data structure of the first processing pipeline.

8. The apparatus of claim 1 , wherein updating the network weights comprises backpropagating errors through multiple layers of the multi-layer artificial neural network.

9. The apparatus of claim 1 , wherein the at least one processor is programmed to process data in the first processing pipeline using multiple processors and process data in the second processing pipeline using the multiple processors.

10. The apparatus of claim 9 , wherein processing data using multiple processors comprises assigning to each of the multiple processors, at least two layers of the multi-layer artificial neural network, wherein the multiple processors are arranged to pass data from one processor to another processor to enable pipelined parallel processing of data in the first processing pipeline and the second processing pipeline.

11. The apparatus of claim 9 , wherein at least two of the multiple processors are graphics processing units (GPUs).

12. The apparatus of claim 1 , wherein the at least one processor is further programmed to:
determine whether the artificial neural network has been sufficiently trained; and
iteratively calculate the network activations using the first processing pipeline and updating the network weights using the second processing pipeline until it is determined that the artificial neural network has been sufficiently trained.

13. An apparatus configured to train a multi-layer artificial neural network for use in speech recognition, wherein each of the layers in the artificial neural network includes a plurality of nodes, each of the plurality of nodes in a layer being connected to a plurality of nodes in one or more adjacent layers of the artificial neural network, wherein the connections between nodes in the artificial neural network are associated with network weights, the apparatus comprising:
at least one processor programmed to implement a first processing pipeline and a second processing pipeline for training the artificial neural network based, at least in part, on a plurality of speech patterns provided to the artificial neural network as input;
wherein the first processing pipeline is configured to determine for a first speech pattern of the plurality of speech patterns, network activations for the plurality of nodes of the artificial neural network in response to providing the first speech pattern as input to the artificial neural network;
wherein the at least one processor is further programmed to determine based, at least in part, on the network activations and a selection criterion, whether the artificial neural network should be trained on the first speech pattern, wherein the selection criterion comprises a focused attention backpropagation criterion, and wherein determining whether the artificial neural network should be trained on the first speech pattern comprises applying the focused attention back propagation criterion to the network activations; and
wherein the second processing pipeline is configured to update the network weights based, at least in part, on the network activations when it is determined that the artificial neural network should be trained on the first speech pattern.

14. The apparatus of claim 13 , wherein the at least one processor is further programmed to:
determine based, at least in part, on the selection criterion, whether the first speech pattern has already been learned by the artificial neural network; and
copy data corresponding to the first speech pattern from the first processing pipeline to the second processing pipeline when it is determined that the first speech pattern has not already been learned by the artificial neural network.

15. The apparatus of claim 13 , wherein the at least one processor comprises at least two graphics processing units (GPUs), wherein each of the at least two GPUs is assigned to perform calculations for at least two layers of the multi-layer artificial neural network, and wherein the at least two GPUs are arranged to pass data between each other to enable pipelined parallel processing of data in the first processing pipeline and the second processing pipeline.

Assignees

• Nuance Communications Inc

Inventors

• Gemello Roberto
• Mana Franco
• Albesano Dario

Classifications

• H10D64/01328

  by defining the conductor using a sidewall spacer mask, a transformation under a mask or a plating at a sidewall · CPC title

• H10D86/215

  comprising FinFETs · CPC title

• G06N3/0499 Primary

  Feedforward networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/04 Primary

  Architecture, e.g. interconnection topology · CPC title