Gap-aware mitigation of gradient staleness

Patent metadata

Field	Value
Publication number	US-11631035-B2
Application number	US-202017773917-A
Country	US
Kind code	B2
Filing date	Nov 5, 2020
Priority date	Nov 5, 2019
Publication date	Apr 18, 2023
Grant date	Apr 18, 2023

Abstract

Disclosed embodiments are a computing system and a computer-implemented method for distributed training of a machine learning model over a plurality of computing nodes, in a plurality of iterations, characterized by gradient gap based mitigation of the gradient staleness problem. The disclosed method evaluates the staleness of the gradient based on the difference in gradients between a central point, for example an iteration's common starting point, and the points reached by the respective computing node during one or more iterations, and aggregates the update steps from the plurality of computing nodes, while giving more weight to computing nodes having a lesser change in the gradient.

First claim

What is claimed is:

1. A system for improving an accuracy of a machine learning model trained using distributed training over a plurality of computing nodes, comprising:
a master computing node configured to control a training of the machine learning model during a plurality of training iterations, wherein the following is performed in each of the plurality of iterations:
instructing each of a plurality of slave computing nodes to train a respective local copy of the machine learning model, locally stored on each respective slave computing node, by locally computing a respective update step on a plurality of parameters of the respective local copy of the machine learning model, based on at least one locally calculated gradient;
obtaining one or more respective update steps from each of the plurality of slave computing nodes;
creating an updated machine learning model with improved accuracy, by merging respective updated local machine learning models using an aggregation function comprising weights, which are a decreasing function of a change magnitude in at least one gradient at a point indicated by the respective update step; and
instructing at least one of the plurality of slave computing nodes to apply a copy of the updated machine learning model in a following training iteration;

wherein at least one of the weights is a constant divided by a difference between the gradient at a central point and the gradient at a point based on the respective update step; and
wherein the change magnitude is measured between at least one first locally calculated gradient at the central point and at least one second locally calculated gradient at the point indicated by the respective update step, the central point is determined by the aggregation function computed during a previous training iteration.

2. The system of claim 1 , wherein the at least one locally calculated gradient computed by each of the plurality of slave computing nodes is computed by applying a stochastic gradient descent for minimizing a loss function for the respective local copy.

3. The system of claim 1 , wherein the aggregation function is averaging of the update steps.

4. The system of claim 1 , wherein each of the plurality of slave computing nodes uses a subset of a training dataset for calculating a value of the update steps by training the respective local copy.

5. The system of claim 1 , wherein the update step is further based on hyper-parameters comprising learning rate, and the learning rate is adjusted according to a gradient convergence measure.

6. A computer implemented method for improving an accuracy of a machine learning model trained using distributed training over a plurality of computing nodes, the plurality of computer nodes comprising a master computing node configured to control a training of the machine learning model during a plurality of training iterations, wherein the following is performed in each of the plurality of iterations:
instructing each of a plurality of slave computing nodes to train a respective local copy of the machine learning model, locally stored on each respective slave computing node, by locally computing a respective update step on a plurality of parameters of the respective local copy of the machine learning model, based on at least one locally calculated gradient;
obtaining one or more respective update steps from each of the plurality of slave computing nodes;
creating an updated machine learning model with improved accuracy, by merging respective updated local machine learning models using an aggregation function comprising weights, which are a decreasing function of a change magnitude in at least one gradient at a point indicated by the respective update step; and
instructing at least one of the plurality of slave computing nodes to apply a copy of the updated machine learning model in a following training iteration;
wherein at least one of the weights is a constant divided by a difference between the gradient at a central point and the gradient at a point based on the respective update step; and
wherein the change magnitude is measured between at least one first locally calculated gradient at the central point and at least one second locally calculated gradient at the point indicated by the respective update step, the central point is determined by the aggregation function computed during a previous training iteration.

7. The computer implemented method of claim 6 , wherein the at least one locally calculated gradient computed by each of the plurality of slave computing nodes is computed by applying a stochastic gradient descent for minimizing a loss function for the respective local copy.

8. The computer implemented method of claim 6 , wherein the aggregation function is averaging of the update steps.

9. The computer implemented method of claim 6 , wherein each of the plurality of slave computing nodes uses a subset of a training dataset for calculating a value of the update steps by training the respective local copy.

10. The computer implemented method of claim 6 , wherein the update step is further based on hyper-parameters comprising learning rate, and the learning rate is adjusted downwards according to a gradient convergence measure.

11. A computer program product for improving an accuracy of a machine learning model trained using distributed training over a plurality of computing nodes, the computer program product comprising a non-transitory computer readable storage medium having:
first program instructions for instructing each of a plurality of slave computing nodes to train a respective local copy of the machine learning model, locally stored on each respective slave computing node, by locally computing a respective update step on a plurality of parameters of the respective local copy of the machine learning model, based on at least one locally calculated gradient;
second program instructions for obtaining one or more respective update steps from each of the plurality of slave computing nodes;
third program instructions for creating an updated machine learning model with improved accuracy, by merging respective updated local machine learning models using an aggregation function comprising weights, which are a decreasing function of a change magnitude in at least one gradient at a point indicated by the respective update step; and
fourth program instructions for instructing at least one of the plurality of slave computing nodes to apply a copy of the updated machine learning model in a following training iteration;
wherein at least one of the weights is a constant divided by a difference between the gradient at a central point and the gradient at a point based on the respective update step; and
wherein the change magnitude is measured between at least one first locally calculated gradient at the central point and at least one second locally calculated gradient at the point indicated by the respective update step, the central point is determined by the aggregation function computed during a previous training iteration.

Assignees

• Technion Res & Dev Foundation

Inventors

• Schuster Assaf
• Barkai Saar
• Hakimi Ido

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N3/08

  Learning methods · CPC title