Asynchronous optimization for sequence training of neural networks

Patent metadata

Field	Value
Publication number	US-11557277-B2
Application number	US-202117644362-A
Country	US
Kind code	B2
Filing date	Dec 15, 2021
Priority date	Nov 4, 2013
Publication date	Jan 17, 2023
Grant date	Jan 17, 2023

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining, by a first sequence-training speech model, a first batch of training frames that represent speech features of first training utterances; obtaining, by the first sequence-training speech model, one or more first neural network parameters; determining, by the first sequence-training speech model, one or more optimized first neural network parameters based on (i) the first batch of training frames and (ii) the one or more first neural network parameters; obtaining, by a second sequence-training speech model, a second batch of training frames that represent speech features of second training utterances; obtaining one or more second neural network parameters; and determining, by the second sequence-training speech model, one or more optimized second neural network parameters based on (i) the second batch of training frames and (ii) the one or more second neural network parameters.

First claim

What is claimed is:

1. A computer-implemented method when executed on data processing hardware of a parameter server causes the data processing hardware to perform operations comprising:
transmitting a current set of parameter values for a neural network model to a plurality of data processing apparatuses each running a corresponding replica of the neural network model, each data processing apparatus in communication with the data processing hardware of the parameter server and configured to:
obtain a corresponding training utterance of one or more predetermined words spoken by a respective training speaker different than each other respective training speaker that spoke the corresponding training utterance obtained for other ones of the plurality of replicas of the neural network model; and
train, using the current set of parameter values for the neural network model and the corresponding training utterance obtained for the corresponding replica of the neural network model, the corresponding replica of the neural network model to generate corresponding updated parameter values for the neural network model; and

from each data processing apparatus of the plurality of data processing apparatuses, receiving the corresponding updated parameter values for the neural network model.

2. The computer-implemented method of claim 1 , wherein each data processing apparatus is configured to train the corresponding replica of the neural network model by training the corresponding replica of the neural network model in parallel with the other ones of the data processing apparatuses training the corresponding other ones of the plurality of replicas of the neural network model.

3. The computer-implemented method of claim 1 , wherein each data processing apparatus is configured to train the corresponding replica of the neural network model by training the corresponding replica of the neural network model independently from the other ones of the data processing apparatuses training the corresponding other ones of the plurality of replicas of the neural network model.

4. The computer-implemented method of claim 1 , wherein each data processing apparatus is configured to train the corresponding replica of the neural network model by training the corresponding replica of the neural network model asynchronously with respect to the other ones of the data processing apparatuses training the corresponding other ones of the plurality of replicas of the neural network model.

5. The computer-implemented method of claim 1 , wherein each data processing apparatus is configured to train the corresponding replica of the neural network model using stochastic gradient descent optimization.

6. The computer-implemented method of claim 1 , wherein receiving the corresponding updated parameter values for the neural network model comprises receiving the corresponding updated parameter values for the neural network model from the corresponding data processing apparatus without receiving the corresponding training utterance obtained for the corresponding replica of the neural network model from the corresponding data processing apparatus.

7. The computer-implemented method of claim 1 , wherein:
the current set of parameter values for the neural network model comprise a current set of weights for the neural network model; and
the corresponding updated parameter values for the neural network model comprise corresponding updated weights for the neural network model.

8. The computer-implemented method of claim 1 , wherein the neural network model is trained to indicate likelihoods that acoustic feature vectors represent different phonetic units.

9. The computer-implemented method of claim 1 , wherein the current set of parameter values for the neural network model comprise weights and biases of hidden layers of the neural network model.

10. The computer-implemented method of claim 1 , wherein the operations further comprise determining an updated set of parameter values for the neural network model based on the current set of parameter values for the neural network model and the corresponding updated parameter values for the neural network model received from each corresponding data processing apparatus of the plurality of data processing apparatuses.

11. A parameter server comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
transmitting a current set of parameter values for a neural network model to a plurality of data processing apparatuses each running a corresponding replica of the neural network model, each data processing apparatus in communication with the data processing hardware of the parameter server and configured to:
obtain a corresponding training utterance of one or more predetermined words spoken by a respective training speaker different than each other respective training speaker that spoke the corresponding training utterance obtained for other ones of the plurality of replicas of the neural network model; and
train, using the current set of parameter values for the neural network model and the corresponding training utterance obtained for the corresponding replica of the neural network model, the corresponding replica of the neural network model to generate corresponding updated parameter values for the neural network model; and

from each data processing apparatus of the plurality of data processing apparatuses, receiving the corresponding updated parameter values for the neural network model.

12. The system of claim 11 , wherein each data processing apparatus is configured to train the corresponding replica of the neural network model by training the corresponding replica of the neural network model in parallel with the other ones of the data processing apparatuses training the corresponding other ones of the plurality of replicas of the neural network model.

13. The system of claim 11 , wherein each data processing apparatus is configured to train the corresponding replica of the neural network model by training the corresponding replica of the neural network model independently from the other ones of the data processing apparatuses training the corresponding other ones of the plurality of replicas of the neural network model.

14. The system of claim 11 , wherein each data processing apparatus is configured to train the corresponding replica of the neural network model by training the corresponding replica of the neural network model asynchronously with respect to the other ones of the data processing apparatuses training the corresponding other ones of the plurality of replicas of the neural network model.

15. The system of claim 11 , wherein each data processing apparatus is configured to train the corresponding replica of the neural network model using stochastic gradient descent optimization.

16. The system of claim 11 , wherein receiving the corresponding updated parameter values for the neural network model comprises receiving the corresponding updated parameter values for the neural network model from the corresponding data processing apparatus without receiving the corresponding training utterance obtained for the corresponding replica of the neural network model from the corresponding data processing apparatus.

17. The system of claim 11 , wherein:
the current set of parameter values for the neural network model comprise a current set of weights for the neural network model; and
the corresp

Assignees

• Google Llc

Inventors

• Heigold Georg
• Mcdermott Erik
• Vanhoucke Vincent O
• Senior Andrew W
• Bacchiani Michiel A U

Classifications

• G10L15/183

  using context dependencies, e.g. language models · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G10L15/063 Primary

  Training · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G06N3/0454

  Physics · mapped topic