Meta-knowledge fine tuning method and platform for multi-task language model

Patent metadata

Field	Value
Publication number	US-11354499-B2
Application number	US-202117531813-A
Country	US
Kind code	B2
Filing date	Nov 22, 2021
Priority date	Nov 2, 2020
Publication date	Jun 7, 2022
Grant date	Jun 7, 2022

Abstract

Disclosed is a meta-knowledge fine tuning method and platform for a multi-task language model. The method is to obtain highly transferable shared knowledge, that is, meta-knowledge, on different data sets of tasks of the same category, perform interrelation and mutual reinforcement on the learning processes of the tasks of the same category that correspond to different data sets and are in different domains, so as to improve the fine tuning effect of downstream tasks of the same category on data sets of different domains in the application of the language model, and improve the parameter initialization ability and the generalization ability of a general language model for the tasks of the same category.

First claim

What is claimed is:

1. A meta-knowledge fine tuning method for a multi-task language model, comprising the following stages:
a first stage, calculating the prototypes of cross-domain data sets of tasks of the same category: embedded features of the prototypes of the corresponding domains of the tasks of the category is intensively learned from the data sets of different domains of the tasks of the same category, and the average embedded feature of all input texts of the tasks of the same category in different domains is taken as a corresponding multi-domain category prototype of the tasks of the same category;
a second stage, calculating typical scores of instances: where d self represents the distance between the embedded feature of each instance and d others represents the distance between the embedded feature of each instance and other domain prototypes; and the typical score of each instance is defined as a linear combination of d self and d others ;
and a third stage, a meta-knowledge fine tuning network based on typical scores: the typical scores obtained in the second stage is used as weight coefficients of the meta-knowledge fine tuning network, and a multi-task typical sensitive label classification loss function is designed as a learning objective function of meta-knowledge fine tuning; and the loss function penalizes the labels of the instances of all domains that the language model predicts incorrectly;
wherein in the first stage, D m k represents a set of input texts x i k with a category label m in a k th domain D k of the data set:
D m k ={x i k V( x i k ,y i k )∈ D k ,y i k =m}

where m∈M, M represents a set of all category labels in the data set; and (x i k , y i k ) represents an i th instance in the k th domain;
the category prototype c m k represents the average embedded feature of all input texts with the category label m in the k th domain:

c
m
k

=

1

D
m
k

⁢

∑

x
i
k

∈

D
m
k

⁢

E
⁡

(

x
i
k

)

wherein, ε(·) represents an embedded expression of x i k output by a BERT model; and for the BERT model, the average embedded feature is the average pooling of the last layer of Transformer encoder corresponding to the input x i k.

2. The meta-knowledge fine tuning method for the multi-task language model according to claim 1 , wherein in the second stage, the typical score t i k of the instance (x i k , y i k ) is expressed as:

t
i
k

=

α
⁢

∑

m
∈
M

⁢

β
m

⁢

cos
⁡

(

E
⁡

(

x
i
k

)

,

c
m
k

)

∑

m
∈
M

⁢

β
m

+

1
-
α

K
-
1

·

∑

k
=
1

K

⁢

1

(

k
~

≠
k

)

⁢

∑

m
∈
M

⁢

β
m

⁢

cos
⁡

(

E
⁡

(

x
i

Assignees

• Zhejiang Lab

Inventors

• Wang Hongsheng
• Shan Haijun
• Hu Shengjian

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title