Technologies for optimized machine learning training

Patent metadata

Field	Value
Publication number	US-10963783-B2
Application number	US-201715436841-A
Country	US
Kind code	B2
Filing date	Feb 19, 2017
Priority date	Feb 19, 2017
Publication date	Mar 30, 2021
Grant date	Mar 30, 2021

Abstract

Technologies for optimization of machine learning training include a computing device to train a machine learning network with a training algorithm that is configured with configuration parameters. The computing device may perform many training instances in parallel. The computing device captures a time series of partial accuracy values from the training. Each partial accuracy value is indicative of machine learning network accuracy at an associated training iteration. The computing device inputs the configuration parameters to a feed-forward neural network to generate a representation and inputs the representation to a recurrent neural network. The computing device trains the feed-forward neural network and the recurrent neural network against the partial accuracy values. The computing device optimizes the feed-forward neural network and the recurrent neural network to determine optimized configuration parameters. The optimized configuration parameters may minimize training time to achieve a predetermined accuracy level. Other embodiments are described and claimed.

First claim

The invention claimed is:

1. A computing device for optimization of machine learning training, the computing device comprising:
a network trainer to (i) train a machine learning network with a training algorithm, wherein the training algorithm is configured with one or more configuration parameters, and (ii) capture a time series of partial accuracy values in response to training of the machine learning network, wherein the time series comprises a plurality of partial accuracy values for a sequence of training iterations performed with the one or more configuration parameters, wherein each partial accuracy value is indicative of machine learning network accuracy at an associated training iteration of the sequence of training iterations;
a network modeler to (i) input the one or more configuration parameters to a feed-forward neural network to generate a representation of the configuration parameters, and (ii) input the representation of the configuration parameters to a recurrent neural network;
a model trainer to train the recurrent neural network and the feed-forward neural network against the time series of partial accuracy values to predict partial accuracy values based on configuration parameters of the machine learning network in response to inputting of the one or more configuration parameters; and
an optimizer to, in response to training of the recurrent neural network and the feed-forward neural network, optimize the recurrent neural network and the feed-forward neural network to determine one or more optimized configuration parameters;
wherein the network trainer is further to train the machine learning network with the training algorithm in response to optimization of the recurrent neural network and the feed-forward neural network, wherein the training algorithm is configured with the one or more optimized configuration parameters.

2. The computing device of claim 1 , wherein the machine learning network comprises a convolutional neural network.

3. The computing device of claim 1 , wherein to optimize the recurrent neural network and the feed-forward neural network comprises to determine the one or more optimized configuration parameters to minimize training time to achieve a predetermined accuracy level.

4. The computing device of claim 3 , wherein to optimize the recurrent neural network and the feed-forward neural network comprises to optimize the recurrent neural network and the feed-forward neural network with a Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm.

5. The computing device of claim 1 , wherein the network trainer is further to train the machine learning network with a plurality of parallel instances of the training algorithm, wherein each parallel instance of the training algorithm is configured with a different set of one or more configuration parameters.

6. The computing device of claim 5 , wherein the network trainer is further to capture a time series of partial accuracy values for each parallel instance of the training algorithm.

7. The computing device of claim 1 , wherein the network trainer is further to train the machine learning network with the training algorithm in parallel by a plurality of distributed computing nodes, wherein the training algorithm of each computing node is configured with a different set of one or more configuration parameters.

8. The computing device of claim 1 , wherein the network trainer is further to capture a time series of partial accuracy values in response to training of the machine learning network with the training algorithm configured with the one or more optimized configuration parameters.

9. The computing device of claim 1 , wherein the feed-forward neural network comprises a deep neural network including a plurality of fully connected layers.

10. The computing device of claim 1 , wherein the recurrent neural network comprises a long short time memory network.

11. A method for optimization of machine learning training, the method comprising:
training, by a computing device, a machine learning network with a training algorithm, wherein the training algorithm is configured with one or more configuration parameters;
capturing, by the computing device, a time series of partial accuracy values in response to training the machine learning network, wherein the time series comprises a plurality of partial accuracy values for a sequence of training iterations performed with the one or more configuration parameters, wherein each partial accuracy value is indicative of machine learning network accuracy at an associated training iteration of the sequence of training iterations;
inputting, by the computing device, the one or more configuration parameters to a feed-forward neural network to generate a representation of the configuration parameters;
inputting, by the computing device, the representation of the configuration parameters to a recurrent neural network;
training, by the computing device, the recurrent neural network and the feed-forward neural network against the time series of partial accuracy values to predict partial accuracy values based on configuration parameters of the machine learning network in response to inputting the one or more configuration parameters;
optimizing, by the computing device in response to training the recurrent neural network and the feed-forward neural network, the recurrent neural network and the feed-forward neural network to determine one or more optimized configuration parameters; and
training, by the computing device, the machine learning network with the training algorithm in response to optimizing the recurrent neural network and the feed-forward neural network, wherein the training algorithm is configured with the one or more optimized configuration parameters.

12. The method of claim 11 , wherein optimizing the recurrent neural network and the feed-forward neural network comprises determining the one or more optimized configuration parameters to minimize training time to achieve a predetermined accuracy level.

13. The method of claim 11 , further comprising training, by the computing device, the machine learning network with a plurality of parallel instances of the training algorithm, wherein each parallel instance of the training algorithm is configured with a different set of one or more configuration parameters.

14. The method of claim 13 , further comprising capturing, by the computing device, a time series of partial accuracy values for each parallel instance of the training algorithm.

15. The method of claim 11 , further comprising training the machine learning network with the training algorithm in parallel by a plurality of distributed computing nodes, wherein the training algorithm of each computing node is configured with a different set of one or more configuration parameters.

16. The method of claim 11 , further comprising capturing, by the computing device, a time series of partial accuracy values in response to training the machine learning network with the training algorithm configured with the one or more optimized configuration parameters.

17. One or more non-transitory, computer-readable storage media comprising a plurality of instructions that in response to being executed cause a computing device to:
train a machine learning network with a training algorithm, wherein the training algorithm is configured with one or more configuration parameters;
capture a time series of partial accuracy values in response to training the machine learning network, wherein the time series comprises a plurality of partial accuracy values for a sequence of training itera

Assignees

• Intel Corp

Inventors

• Faivishevsky Lev
• Armon Amitai

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title