Fast converging gradient compressor for federated learning

Patent metadata

Field	Value
Publication number	US-12361315-B2
Application number	US-202117154243-A
Country	US
Kind code	B2
Filing date	Jan 21, 2021
Priority date	Jan 21, 2021
Publication date	Jul 15, 2025
Grant date	Jul 15, 2025

Abstract

Method for model updating in a federated learning environment, including distributing a current model to client nodes; receiving a first set of gradient sign vectors, wherein each gradient sign vector of the first set of gradient sign vectors is received from one client node; generating a first updated model based on the first set of gradient sign vectors; distributing the first updated model to the plurality of client nodes; storing a first shape parameter and a second shape parameter; receiving, in response to distributing the first updated model, a second set of gradient sign vectors, wherein each gradient sign vector of the second set of gradient sign vectors is received from one client node; generating a second updated model based on the second set of gradient sign vectors, the first shape parameter, and the second shape parameter; and distributing the second updated model to the plurality of client nodes.

First claim

What is claimed is:

1. A method for model updating in a federated learning environment, the method comprising:
distributing, by a model coordinator, a current model with index positions to a plurality of physical client nodes comprising a plurality of neural networks, wherein the current model is trained on the plurality of neural networks using private data in the plurality of physical client nodes producing a gradient vector in each of the plurality of physical client nodes;
receiving, by the model coordinator and in response to distributing the current model, a first set of gradient sign vectors, wherein;
each gradient sign vector of the first set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes,
each gradient sign vector of the first set of gradient sign vectors is based on a sign of the gradient vector for each of the index positions generated by respective physical client nodes of the plurality of physical client nodes using the current model, and
each gradient sign vector initially has a value of zero if the gradient vector is a negative number and a value of one if the gradient vector is zero or a positive number to compress an amount of information transmitted to the model coordinator from each physical client node by compressing each index position to a single bit, which reduces a network bandwidth necessary to transmit the first set of gradient sign vectors;

generating, by the model coordinator, a first updated model by using a Bayesian Beta-Bernoulli model to determine an expected value for each of the index positions with inputs comprising the first set of gradient sign vectors, a first shape parameter, and a second shape parameter;
distributing the first updated model to the plurality of physical client nodes;
storing, by the model coordinator, the first shape parameter and the second shape parameter;
receiving, by the model coordinator and in response to distributing the first updated model, a second set of gradient sign vectors, wherein each gradient sign vector of the second set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes;
generating, by the model coordinator, a second updated model based on the second set of gradient sign vectors, the first shape parameter, and the second shape parameter; and
distributing the second updated model to the plurality of physical client nodes.

2. The method of claim 1 , further comprising:
updating the first shape parameter using a set of gradient sign values at a gradient sign vector index position within the second set of gradient sign vectors to obtain an updated first shape parameter in a Bayesian update;
updating the second shape parameter using the set of gradient sign values at the gradient sign vector index position within the second set of gradient sign vectors to obtain an updated second shape parameter in a Bayesian update;
receiving, by the model coordinator and in response to distributing the second updated model, a third set of gradient sign vectors, wherein each gradient sign vector of the third set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes; and
generating, by the model coordinator, a third updated model using the second set of gradient sign vectors, the updated first shape parameter, and the updated second shape parameter.

3. The method of claim 2 , wherein:
updating the first shape parameter using the set of gradient sign values comprises adding a quantity of positive gradient sign values of the set of gradient sign values to the first shape parameter, and
updating the second shape parameter using the set of gradient sign values comprises adding a quantity of negative gradient sign values of the set of gradient sign values to the second shape parameter.

4. The method of claim 2 , further comprising:
making a determination, by the model coordinator and after distributing the second updated model, that a cycle threshold is reached;
discarding, based on the determination, the updated first shape parameter and the updated second shape parameter; and
using the first shape parameter, the second shape parameter, and a next set of gradient vectors from the plurality of physical client nodes to generate the next updated model.

5. The method of claim 1 , wherein, when the current model is an initial model, generating the first updated model comprises:
calculating an average gradient index position value for each gradient index position of the first updated model using the first set of gradient sign vectors.

6. The method of claim 1 , further comprising, before generating the first updated model:
decoding each gradient sign vector of the first set of gradient sign vectors to include only non-zero real numbers to obtain a set of decoded gradient sign vectors,
wherein the model coordinator generates the first updated model using the set of decoded gradient sign vectors.

7. The method of claim 1 , wherein the first shape parameter and the second shape parameter are equal to one.

8. The method of claim 1 , wherein prior to generating a first updated model, each gradient sign vector of the first set of gradient sign vectors is decoded where a value of zero is changed to negative one and a value of one is changed to positive one.

9. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for model updating in a federated learning environment, the method comprising:
distributing, by a model coordinator, a current model with index positions to a plurality of physical client nodes comprising a plurality of neural networks, wherein the current model is trained on the plurality of neural networks using private data in the plurality of physical client nodes producing a gradient vector in each of the plurality of physical client nodes;
receiving, by the model coordinator and in response to distributing the current model, a first set of gradient sign vectors, wherein:
each gradient sign vector of the first set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes,
each gradient sign vector of the first set of gradient sign vectors is based on a sign of the gradient vector for each of the index positions generated by respective physical client nodes of the plurality of physical client nodes using the current model, and
each gradient sign vector initially has a value of zero if the gradient vector is a negative number and a value of one if the gradient vector is zero or a positive number to compress an amount of information transmitted to the model coordinator from each physical client node by compressing each index position to a single bit, which reduces a network bandwidth necessary to transmit the first set of gradient sign vectors;

generating, by the model coordinator, a first updated model by using a Bayesian Beta-Bernoulli model to determine an expected value for each of the index positions with inputs comprising the first set of gradient sign vectors, a first shape parameter, and a second shape parameter;
distributing the first updated model to the plurality of physical client nodes;
storing, by the model coordinator, the first shape parameter and the second shape parameter;
receiving, by the model coordinator and in response to distributing the first updated model, a second set of gradient sign vectors, wherein each gradient sign vector of the second set of gradient sign vectors is received from one physical client node of the plurality of physical client nodes;
generating, by the model coor

Assignees

• Emc Ip Holding Co Llc

Inventors

• Ferreira Paulo Abelha
• Da Silva Pablo Nascimento
• Calmon Tiago Salviano
• Nery Stelling Neto Roberto
• Gottin Vinicius Michel

Classifications

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• H04L67/10

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title