Federated ensemble learning from decentralized data with incremental and decremental updates

Patent metadata

Field	Value
Publication number	US-2022121999-A1
Application number	US-202017073341-A
Country	US
Kind code	A1
Filing date	Oct 17, 2020
Priority date	Oct 17, 2020
Publication date	Apr 21, 2022
Grant date	—

Abstract

A computer implemented method includes distributing a plurality of prediction models, where each of a plurality of clients initially includes at least one associated prediction model from the plurality of prediction models, among all of the plurality of clients to provide each of the plurality of clients with each of the plurality of prediction models. The plurality of prediction models is evaluated on at least a portion of a local dataset resident on each of the plurality of clients to output a quantification indicating how each of the prediction models fit at least the portion of the local dataset of each of the plurality of clients. An ensemble model is generated by applying weights to each of the plurality of prediction models based on a value, a gradient, and a Hessian matrix of a user-defined objective.

First claim

What is claimed is:

1 . A computer implemented method comprising:
distributing a plurality of prediction models, where each of a plurality of clients initially includes at least one associated prediction model from the plurality of prediction models, among all of the plurality of clients to provide each of the plurality of clients with each of the plurality of prediction models; evaluating each of the plurality of prediction models on at least a portion of a local dataset resident on each of the plurality of clients to output a quantification indicating how each of the prediction models fit at least the portion of the local dataset of each of the plurality of clients; and generating an ensemble model by applying weights to each of the plurality of prediction models based on a value, a gradient, and a Hessian matrix of a user-defined objective.

2 . The computer implemented method of claim 1 , wherein the ensemble model is generated in a decentralized manner without including an exchange of raw data among the plurality of clients.

3 . The computer implemented method of claim 1 , wherein the vector is independent of a size of each of the datasets resident on each of the plurality of clients.

4 . The computer implemented method of claim 1 , further comprising performing one or more rounds of distributed gradient descent on the weights.

5 . The computer implemented method of claim 1 , further comprising limiting a number of models from the plurality of models that are assigned a weight greater than zero to a predetermined threshold.

6 . The computer implemented method of claim 1 , further comprising excluding a number of clients, and additional associated prediction models resident on respective ones of the number of clients, from the plurality of clients.

7 . The computer implemented method of claim 1 , further comprising:
adding an additional client to the plurality of clients; distributing each of the prediction models of the plurality of prediction models to the additional client; and distributing one or more additional models associated with the additional client to the plurality of clients such that the additional model is evaluated on datasets resident on each of the plurality of clients, and each of the plurality of models, including the one or more additional models, are evaluated on the additional client.

8 . The computer implemented method of claim 1 , further comprising, removing or ignoring the associated prediction model of a removed client from each of the plurality of clients.

9 . The computer implemented method of claim 1 , further comprising, upon determining that a model has changed on one of the plurality of clients, re-valuating the changed model on each of the plurality of clients on at least the portion of the dataset resident on each of the plurality of clients.

10 . The computer implemented method of claim 1 , further comprising, upon determining that at least the portion of the dataset of a changed client of the plurality of clients is changed, re-evaluating each of the plurality of models on at least the portion of the dataset of the changed client.

11 . The computer implemented method of claim 1 , further comprising optimizing the weights applied to each of the plurality of models to minimize error between a predicted label given by the ensemble model and a ground truth label.

12 . The computer implemented method of claim 1 , further comprising sending to a central server, each vector for each of the plurality of clients.

13 . A computerized federated ensemble learning system comprising:
a plurality of clients in communication with a server; one or more prediction models resident at each of the plurality of clients; and a dataset resident at each of the plurality of clients, wherein: each of the one or more prediction models are distributed among each of the plurality of clients; each of the plurality of clients is configured to evaluate each of the plurality of prediction models on at least a portion of the dataset resident on each of the plurality of clients and output a quantification indicating how each of the prediction models fit at least the portion of the local dataset of each of the plurality of clients; and the server is configured to receive the vector from each of the plurality of clients and generate an ensemble model by applying weights to each of the plurality of prediction models.

14 . The computerized federated ensemble learning system of claim 13 , wherein the ensemble model is generated in a decentralized manner without including an exchange of raw data among the plurality of clients.

15 . The computerized federated ensemble learning system of claim 13 , wherein the vector is independent of a size of each dataset resident at each of the plurality of clients.

16 . The computerized federated ensemble learning system of claim 13 , wherein the server is configured to assign the weights such that a predetermined threshold number of models from the plurality of models are assigned a weight greater than zero.

17 . A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out a method of improving computing efficiency of a computing device operating a federated learning system, the method comprising:
distributing a plurality of prediction models, where each of a plurality of clients initially includes at least one associated prediction model from the plurality of prediction models, among all of the plurality of clients to provide each of the plurality of clients with each of the plurality of prediction models; evaluating each of the plurality of prediction models on at least a portion of a local dataset resident on each of the plurality of clients to output a quantification indicating how each of the prediction models fit at least the portion of the local dataset of each of the plurality of clients; and generating an ensemble model by applying weights to each of the plurality of prediction models based on a value, a gradient, and a Hessian matrix of a user-defined objective.

18 . The non-transitory computer readable storage medium of claim 17 , wherein the vector is independent of a size of each dataset resident on each of the plurality of clients.

19 . The non-transitory computer readable storage medium of claim 17 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising limiting a number of models from the plurality of models that are assigned a weight greater than zero to a predetermined threshold.

20 . The non-transitory computer readable storage medium of claim 17 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising optimizing the weights applied to each of the plurality of models to minimize error between a predicted label given by the ensemble model and a ground truth label.

Assignees

• IBM

Inventors

• Wang Shiqiang
• Chakraborty Supriyo
• Desai Nirmit V
• Freimuth Douglas M
• Lee Wei-Han
• Liu Changchang

Classifications

• G06N20/20 Primary

  Ensemble learning · CPC title

• G06N5/04

  Inference or reasoning models · CPC title