Two-phase distributed neural network training system

Patent metadata

Field	Value
Publication number	US-10360500-B2
Application number	US-201815946331-A
Country	US
Kind code	B2
Filing date	Apr 5, 2018
Priority date	Apr 20, 2017
Publication date	Jul 23, 2019
Grant date	Jul 23, 2019

Abstract

A computing system provides distributed training of a neural network model. Explore phase options, exploit phase options, a subset of a training dataset, and a validation dataset are distributed to a plurality of computing devices. (a) Execution of the model by the computing devices is requested using the subset stored at each computing device. (b) A first result of the execution is received from a computing device. (c) Next configuration data for the neural network model is selected based on the first result and distributed to the computing device. (a) to (c) is repeated until an exploration phase is complete. (d) Execution of the neural network model is requested. (e) A second result is received. (f) Next configuration data is computed based on the second result and distributed to the computing device. (d) to (f) is repeated until an exploitation phase is complete. The next configuration data defines the model.

First claim

What is claimed is:

1. A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a first computing device cause the first computing device to:
distribute explore phase optimization parameter values to each computing device of a plurality of computing devices;
distribute a subset of a training dataset to each computing device of the plurality of computing devices;
distribute a validation dataset to each computing device of the plurality of computing devices;
request initialization of a neural network model by the plurality of computing devices;
(a) request execution of the initialized neural network model by the plurality of computing devices using the subset of the training dataset stored at each computing device of the plurality of computing devices;
(b) receive a first result of the execution from a computing device of the plurality of computing devices, wherein the first result is an accuracy value computed using the distributed validation dataset;
(c) identify a computing device of the plurality of computing devices that has an extremum value for the accuracy value in comparison to other computing devices of the plurality of computing devices;
(d) request next configuration data for the neural network model from the identified computing device;
(e) receive the requested next configuration data from the identified computing device;
(f) distribute the received next configuration data to the plurality of computing devices except the identified computing device;
repeat (a) to (f) until an exploration phase is complete based on the explore phase optimization parameter values, wherein in (a), the neural network model is reinitialized with the distributed next configuration data at each computing device of the plurality of computing devices;
distribute exploit phase optimization parameter values to each computing device of the plurality of computing devices;
(g) request execution of the neural network model by the plurality of computing devices using the subset of the training dataset stored at each computing device of the plurality of computing devices;
(h) receive a second result of the execution from the computing device of the plurality of computing devices;
(i) compute next configuration data for the neural network model based on the received second result;
(j) distribute the computed next configuration data to the computing device;
repeat (g) to (j) until an exploitation phase is complete based on the exploit phase optimization parameter values; and
output at least a portion of the computed next configuration data to define a trained neural network model.

2. The non-transitory computer-readable medium of claim 1 , wherein each distributed subset is different.

3. The non-transitory computer-readable medium of claim 1 , wherein the validation dataset distributed to each computing device is identical.

4. The non-transitory computer-readable medium of claim 1 , wherein initialization of the neural network model at each computing device is identical.

5. The non-transitory computer-readable medium of claim 1 , wherein the first computing device is not included in the plurality of computing devices.

6. The non-transitory computer-readable medium of claim 1 , wherein the explore phase optimization parameter values include at least one value that is not equal to a corresponding value of the exploit phase optimization parameter values.

7. The non-transitory computer-readable medium of claim 1 , wherein (a) to (f) are repeated asynchronously for each computing device of the plurality of computing devices.

8. The non-transitory computer-readable medium of claim 7 , wherein (g) to (j) are repeated synchronously for the plurality of computing devices.

9. The non-transitory computer-readable medium of claim 1 , wherein (a) to (f) are repeated synchronously for the plurality of computing devices.

10. The non-transitory computer-readable medium of claim 9 , wherein (g) to (j) are repeated asynchronously for each computing device of the plurality of computing devices.

11. The non-transitory computer-readable medium of claim 1 , wherein the received second result is a gradient vector computed using the distributed validation dataset.

12. The non-transitory computer-readable medium of claim 11 , wherein computing the next configuration data for the neural network model based on the received second result comprises:
solving an optimization problem defined by the exploit phase optimization parameter values using the gradient vector from each computing device of the plurality of computing devices;
performing a collective reduce operation to aggregate the gradient vector from each computing device of the plurality of computing devices; and
computing new neural network weight values from the aggregated gradient vector, wherein the next configuration data includes the new neural network weight values.

13. The non-transitory computer-readable medium of claim 1 , wherein distributing the selected next configuration data in (f) comprises sending, by the first computing device, the configuration data for the neural network model computed by the identified computing device to remaining devices of the plurality of computing devices.

14. The non-transitory computer-readable medium of claim 1 , wherein distributing the selected next configuration data in (f) comprises requesting that the identified computing device send configuration data for the neural network model computed by the identified computing device to remaining devices of the plurality of computing devices.

15. The non-transitory computer-readable medium of claim 1 , wherein identifying the computing device in (c) comprises:
determining if the accuracy value is better than a current accuracy value; and
when the accuracy value is better than a current accuracy value, the next configuration data for the neural network model is requested from the computing device and stored as the next configuration data and the current accuracy value is set equal to the accuracy value.

16. The non-transitory computer-readable medium of claim 15 , wherein distributing the selected next configuration data in (f) comprises sending the stored next configuration data to the computing device when the accuracy value is not better than the current accuracy value.

17. The non-transitory computer-readable medium of claim 15 , wherein the accuracy value is better than the current accuracy value when the accuracy value is greater than the current accuracy value.

18. The non-transitory computer-readable medium of claim 15 , wherein after the exploration phase is complete and before (g), the computer-readable instructions further cause the first computing device to send the stored next configuration data to each computing device of the plurality of computing devices to reinitialize the neural network model at each computing device of the plurality of computing devices.

19. The non-transitory computer-readable medium of claim 1 , wherein execution of the initialized neural network model by the plurality of computing devices in (a) comprises second computer-readable instructions that when executed by each computing device of the plurality of computing devices cause each computing device of the plurality of computing devices to:
(aa) select a set of observation vectors from the subset of the training dataset stored at each computing device;
(ab) execute the neural network model using the distributed explore phase optimization parameter values with the selecte

Assignees

• Sas Inst Inc

Inventors

• Kabul Mustafa Onur
• Lewis Lawrence E

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F9/46

  Multiprogramming arrangements · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/08 Primary

  Learning methods · CPC title