Federated large model adaptive learning system

Patent metadata

Field	Value
Publication number	US-2025103952-A1
Application number	US-202318521883-A
Country	US
Kind code	A1
Filing date	Nov 28, 2023
Priority date	Sep 25, 2023
Publication date	Mar 27, 2025
Grant date	—

Abstract

The present invention provides a federated large model adaptive learning system. Based on the combination of multiobjective optimization and incremental learning, multiple optimization indexes are constructed, and adaptive mini model incremental learning is designed. A gradient scaling method of mini models is proposed for data privacy protection under federated learning, to make full use of gradient information. A correlation between the generalization ability and sampling data is revealed to propose a generalization ability evaluation function. With respect to the real problems of performance degradation and fault faced by industrial equipment during operation, multiple optimization objectives are designed in combination with the generalization ability evaluation function, and the models are updated and repaired adaptively through multiobjective evolutionary learning, to improve the usability of large models in real industrial scenarios. Finally, the adaptive accurate update of the large models and mini models is realized to improve the generalization ability of the models.

First claim

1 . A federated large model adaptive learning system, which is mainly composed of a mini model adaptive update module, a BERT large model and mini model normalization module, a BERT large model adaptive update module and a system privacy protection module, wherein
(1) mini model adaptive update module for the diffculty in interaction between BERT large models and mini models, the mini model adaptive update module is designed, considering three optimization directions of mini model accuracy, mini model forgetting rate and mini model error, the mini model adaptive update module establishes adaptive criteria through the above optimization directions: 1) mini model accuracy submodule: from the perspective of universality, in the mini model adaptive update, the accuracy of the mini models will determine the universality of the BERT large models, therefore, a mini model accuracy submodule is proposed, expressed as follows:

max
⁢

C
m

=

1
m

⁢

∑

i
=
1

m

t
i

 wherein C m represents the average accuracy value of the mini model after an m incremental stage, and t i represents the accuracy value corresponding to an i stage,
2) mini model forgetting rate submodule:
from the perspective of functionality, in the mini model adaptive update, the mini model forgetting rate determines the convergence property of the mini models, and further determines the convergence of the BERT large models, therefore, a mini model forgetting rate submodule is designed, expressed as follows:

min
⁢

F
⁡
(
q
)

=

ud

+

(

1
-
u

)

⁢

F
⁡
(

q
-
1

)

 wherein F(q) represents the mini model forgetting rate of the mini model at time q; F(q) represents the mini model forgetting rate of the mini model at time q−1, u is a coefficient with a value range between [0,1], which is used to control a decay rate, and d is data at current time;
3) mini model error gradient submodule:
from the perspective of high efficiency, in the mini model adaptive update, the error gradient directly determines the effciency of the BERT models, therefore, a mini model error gradient submodule is designed, expressed as follows:

min
⁢

w
⁡
(
v
)

=

w
⁡
(

v
-
1

)

+

η
×

∇

E
⁡
(
v
)

 wherein w(v) represents the weight at time v, w(v−1) represents the weight at time v−1, η is a learning rate, and ∇E(v) represents the error gradient at time v;
(2) BERT large model and mini model normalization module
in the process of the mini model adaptive update, mini model gradient information is generated continuously; therefore, the BERT large model and mini model normalization module will realize normalization of the BERT large modules and the mini models in the gradient information to establish a basis for the implementation of the BERT large model adaptive update module; the gradient information has two properties of size and direction; the privacy protection principle of federated learning allows the mini models to transmit the gradient information to the BERT large models, and the gradient information generated by the mini models is used for feedback learning of the BERT large models; however, there is a huge difference in the number of parameters between the large BERT models and the mini models, and the gradient of the mini models cannot be directly used by the BERT large models; therefore; a method based on gradient scaling is proposed to assess the difference in the number of param,ters between the mini models and the BERT large models and priori knowledge, so as to establish the corresponding relationship between the gradient values of the mini models and the gradient values of the BERT large models; the gradient scaling method is expressed as follows:

T
grad
′

=

t
grad
′

(

T
grad

2
⁢

t

grad

+

T
n

2
⁢

t
n

)

 wherein T grad ′ represents the corresponding gradient value of the mini models on the large models, t grad ′ represents the gradient value of the mini models, T grad is the priori gradient value of the large models, t grad is the priori gradient value of the mini models, T n is the number of parameters of the large models, and T n is the number of parameters of the mini models;
(3) BERT large model adaptive update module
in the BERT large model and mini model normalization module, the normalization of the mini model gradient inf

Assignees

• Univ Hebei Technology
• Zhejiang Lab
• Shenzhen Institute For Advanced Study Univ Of Electronic Science And Technology Of China
• Univ Dalian Tech

Inventors

• Cao Bin
• Jiang Zeyu
• Liu Xin
• Cheng Wen
• Li Yun
• Shen Rensheng
• Chang Yuchun

Classifications

• G06N3/08

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• Y02T10/40

  Engine management systems · CPC title

• G06F21/6245

  Protecting personal data, e.g. for financial or medical purposes · CPC title