Method and device for parallel processing in model training

Patent metadata

Field	Value
Publication number	US-9508347-B2
Application number	US-201314108237-A
Country	US
Kind code	B2
Filing date	Dec 16, 2013
Priority date	Jul 10, 2013
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

A method and a device for training a DNN model includes: at a device including one or more processors and memory: establishing an initial DNN model; dividing a training data corpus into a plurality of disjoint data subsets; for each of the plurality of disjoint data subsets, providing the data subset to a respective training processing unit of a plurality of training processing units operating in parallel, wherein the respective training processing unit applies a Stochastic Gradient Descent (SGD) process to update the initial DNN model to generate a respective DNN sub-model based on the data subset; and merging the respective DNN sub-models generated by the plurality of training processing units to obtain an intermediate DNN model, wherein the intermediate DNN model is established as either the initial DNN model for a next training iteration or a final DNN model in accordance with a preset convergence condition.

First claim

What is claimed is:

1. A method of training a Deep Neural Network (DNN) model, comprising:
at a head computing device comprising one or more processors and memory:
establishing an initial DNN model;
dividing a training data corpus into a plurality of disjoint data subsets;
for each of the plurality of disjoint data subsets, providing the data subset to a respective training processing unit of a plurality of training processing units operating in parallel, wherein a training processing unit is provided by a respective leaf computing device of a plurality of leaf computing devices operating in parallel, the plurality of leaf computing devices being coupled to the head computing device, and wherein the respective training processing unit applies a Stochastic Gradient Descent (SGD) process to update the initial DNN model to generate a respective DNN sub-model of a plurality of DNN sub-models based on the data subset; and
merging the respective DNN sub-models generated by the plurality of training processing units to obtain an intermediate DNN model, wherein the intermediate DNN model is established as either a respective initial DNN model for a next training iteration or a final DNN model in accordance with a preset convergence condition, and wherein merging the respective DNN sub-models comprises:
for each DNN sub-model of the plurality of DNN sub-models, assigning a respective first merging weight to each layer of a plurality of layers of each DNN sub-model; and
for the plurality of DNN sub-models, applying a linear combination of the respective DNN sub-models to obtain the intermediate DNN model, wherein applying the linear combination comprises:
assigning a respective second merging weight to each DNN sub-model of the plurality of DNN sub-models, wherein the respective second merging weight of each DNN sub-model is a vector of the respective first merging weights of all layers of the respective DNN sub-model,

wherein the initial and final DNN models are acoustic models of speech recognition and the training data corpus comprises a plurality of randomized speech files.

2. The method of claim 1 , wherein merging the respective DNN sub-models generated by the plurality of training processing units further comprises:
using a respective shared first merging weight for all layers of each DNN sub-model during the merging.

3. This method of claim 1 , further comprising:
identifying a plurality of decoding processing units operating in parallel, each decoding processing units utilizing a respective final DNN model;
providing a same test sample to each of the plurality of decoding processing units operating in parallel, wherein each decoding processing unit generates a respective posterior probability sequence for the same test sample based on the respective final DNN model of the decoding processing unit; and
merging the respective posterior probability sequences generated by the plurality of decoding processing units to obtain a recognition result for the same test sample.

4. The method of claim 3 , wherein merging respective posterior probability sequences generated by the plurality of decoding processing units further comprises:
using a respective shared merging weight for all phoneme binding states of each respective posterior probability sequence during the merging of the respective posterior probability sequences generated by the plurality of decoding processing units.

5. The method of claim 3 , wherein merging respective posterior probability sequences generated by the plurality of decoding processing units further comprises:
using a respective merging weight for each phoneme binding state of each DNN sub-model during the merging of the respective posterior probability sequences generated by the plurality of decoding processing units.

6. A system for training a Deep Neural Network (DNN) model, comprising:
a head computing device; and
a plurality of leaf computing devices operating in parallel and coupled to the head computing device, wherein head computing device comprises:
one or more processors; and
memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising:
establishing an initial DNN model;
dividing a training data corpus into a plurality of disjoint data subsets;
for each of the plurality of disjoint data subsets, providing the data subset to a respective training processing unit of a plurality of training processing units operating in parallel, wherein a training processing unit is provided by a respective leaf computing device of the plurality of leaf computing devices operating in parallel, and wherein the respective training processing unit applies a Stochastic Gradient Descent (SGD) process to update the initial DNN model to generate a respective DNN sub-model of a plurality of DNN sub-models based on the data subset; and
merging the respective DNN sub-models generated by the plurality of training processing units to obtain an intermediate DNN model, wherein the intermediate DNN model is established as either a respective initial DNN model for a next training iteration or a final DNN model in accordance with a preset convergence condition, and wherein merging the respective DNN sub-models comprises:
for each DNN sub-model of the plurality of DNN sub-models, assigning a respective first merging weight to each layer of a plurality of layers of each DNN sub-model; and
for the plurality of DNN sub-models, applying a linear combination of the respective DNN sub-models to obtain the intermediate DNN model, wherein applying the linear combination comprises:
assigning a respective second merging weight to each DNN sub-model of the plurality of DNN sub-models, wherein the respective second merging weight of each DNN sub-model is a vector of the respective first merging weights of all layers of the respective DNN sub-model,

wherein the initial and final DNN models are acoustic models of speech recognition and the training data corpus comprises a plurality of randomized speech files.

7. The system of claim 6 , wherein merging the respective DNN sub-models generated by the plurality of training processing units further comprises:
using a respective shared first merging weight for all layers of each DNN sub-model during the merging.

8. The system of claim 6 , wherein the operations further comprise:
identifying a plurality of decoding processing units operating in parallel, each decoding processing units utilizing a respective final DNN model;
providing a same test sample to each of the plurality of decoding processing units operating in parallel, wherein each decoding processing unit generates a respective posterior probability sequence for the same test sample based on the respective final DNN model of the decoding processing unit; and
merging the respective posterior probability sequences generated by the plurality of decoding processing units to obtain a recognition result for the same test sample.

9. The system of claim 8 , wherein merging respective posterior probability sequences generated by the plurality of decoding processing units further comprises:
using a respective shared merging weight for all phoneme binding states of each respective posterior probability sequence during the merging of the respective posterior probability sequences generated by the plurality of decoding processing units.

10. The system of claim 8 , wherein merging respective posterior probability sequences generated by the plurality of decoding processing units further comprises:
using a respective merging weight for each phoneme binding state of each DNN sub-model during the merging of the resp

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Wang Eryu
• Lu Li
• Zhang Xiang
• Liu Haibo
• Rao Feng
• Li Lou
• Yue Shuai
• Chen Bo

Classifications

• G10L15/16

  using artificial neural networks · CPC title

• G10L15/34 Primary

  Adaptation of a single recogniser for parallel processing, e.g. by use of multiple processors or cloud computing · CPC title

• G10L15/063

  Training · CPC title

• G06N3/02

  Neural networks · CPC title