Multi-task learning incorporating dependencies method for bionic eye's face attribute recognition

Patent metadata

Field	Value
Publication number	US-11037035-B2
Application number	US-201916431546-A
Country	US
Kind code	B2
Filing date	Jun 4, 2019
Priority date	Jun 4, 2019
Publication date	Jun 15, 2021
Grant date	Jun 15, 2021

Abstract

The application discloses a multi-task learning incorporating dependencies method for bionic eye's face attribute recognition, which is as follows: Determine the first face attribute and the second face attribute for attribute recognition of facial image. Obtain the first recognition task branch and the second recognition task branch. Establish the task dependency between the first recognition task branch and the second recognition task branch to obtain the first transformed face attribute fully connected layer related to the second face attribute. and the second transformed face attribute fully connected layer related to the first face attribute. Feed the first transformed face attribute fully connected layer into the prediction layer to predict the first face attribute of facial image. And feed the second transformed face attribute fully connected layer into the prediction layer to predict the second face attribute of facial image. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition will be obtained according to the above steps.

First claim

What is claimed is:

1. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition including:
determine a first face attribute and a second face attribute for attribute recognition of facial image,
according to the first face attribute and the second face attribute respectively get a first recognition task branch and a second recognition task branch,
establish the task dependency between the first recognition task branch and the second recognition task branch, including: generate the first attention module corresponding to the first face attribute and the second attention module corresponding to the second face attribute; deal with the first fully connected layer in accordance with the second attention module to obtain the first face attribute representation unit related to the second face attribute; and deal with the second fully connected layer in accordance with the first attention module to obtain the second face attribute representation unit related to the first face attribute,
the first transformed face attribute fully connected layer related to the second face attribute is obtained in the first recognition task branch according to the task dependency, and
the second transformed face attribute fully connected layer related to the first face attribute is obtained in the second recognition task branch according to the task dependency,
feed the first transformed face attribute fully connected layer into a prediction layer to predict the first face attribute of the facial image, and feed the second transformed face attribute fully connected layer into the prediction layer to predict the second face attribute of the facial image,
convolutional neural network for attribute recognition of facial image is obtained in accordance with the above steps.

2. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition according to claim 1 , wherein the first recognition task branch and the second recognition task branch gained according to the first face attribute and the second face attribute, which includes:
determine the sharing layer applicable to all face attributes in the baseline network used for attribute recognition,
determine the first residual block and the second residual block corresponding to the first face attribute and the second face attribute in baseline network, and then connect them to the sharing layer,
determine the first fully connected layer and the second fully connected layer corresponding to the first face attribute and the second face attribute,
connect the first and second fully connected layers separately to the first and second residual blocks.

3. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition according to claim 1 , wherein the first fully connected layer is dealt with in accordance with the second attention module to obtain the first face attribute representation unit related to the second face attribute, which includes:
feed all the first face attribute representation units in the first fully connected layer and the i-th second face attribute representation unit in the second fully connected layer into the i-th second face attribute attention module in the second attention module to learn the i-th first face attribute representation unit related to the second face attribute.

4. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition in claim 1 , wherein the second fully connected layer is dealt with in accordance with the first attention module to obtain the second face attribute representation unit related to the first face attribute, which includes:
feed all the second face attribute representation units in the second fully connected layer and the i-th first face attribute representation unit in the first fully connected layer into the i-th first face attribute attention module in the first attention module to learn the i-th second face attribute expression unit related to the first face attribute.

5. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition according to claim 3 , wherein is the first transformed facial attribute fully connected layer related to the said second face attribute is obtained in accordance with task dependency in the first recognition task branch, which includes:
concatenate the first face attribute representation units related to all the second face attributes to generate the first transformed face attribute fully connected layer.

6. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition according to claim 4 , wherein the second transformed facial attribute fully connected layer related to the first face attribute is obtained in accordance with task dependency in the second recognition task branch, which includes:
concatenate the second face attribute representation units related to all the first face attributes to generate the second transformed face attribute fully connected layer.

7. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition according to claim 5 , wherein the first transformed face attribute fully connected layer is fed into the prediction layer to predict the first face attribute of facial images, which includes:
predict the first face attribute by feeding the first transformed face attribute fully connected layer into softmax layer, and then obtain the first face attribute prediction probability.

8. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition according to claim 6 , wherein the second transformed face attribute fully connected layer is fed into the prediction layer to predict the second face attribute of facial images, which includes:
predict the second face attribute by feeding the second transformed face attribute fully connected layer into softmax layer, and then obtain the second face attribute prediction probability.

9. The multi-task learning incorporating dependencies method for bionic eye's face attribute recognition according to claim 3 , wherein all the first face attribute representation units in the first fully connected layer and the i-th second face attribute representation unit in the second fully connected layer are entered into the i-th second face attribute attention module in the second attention module to learn the i-th first face attribute representation unit related to the second face attribute, which include:
score the relevance between the j-th first face attribute representation unit x sj in the first fully connected layer FCs and the i-th second face attribute context unit C Gi in the second fully connected layer in accordance with the following scoring function:
score( x S j ,C G i )=tanh( W S x S j +W G C G i ),

in which, the i-th second face attribute context unit C Gi in the second fully connected layer refers to the i-th second face attribute expression unit x Gi in the second fully connected layer,
use probability P(d 1 =j|x s ,C G i ) to show the relative importance of x sj based on C Gi , in which d 1 refers to the importance of x s based on C Gi of all the first face attribute representation units, in which P(d 1 =j|X S ,C G i ) is calculated using relevance scoring function in accordance with the following equation:

P
⁡

(

Assignees

• Beijing Institute Tech

Inventors

• Huang Qiang
• Chen Xiaopeng
• Fan Di
• Kim Hyunwoo
• Chen Xuechao

Classifications

• G06V40/174

  Facial expression recognition · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V40/171 Primary

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title

• G06V30/194

  References adjustable by an adaptive method, e.g. learning · CPC title

• G06V10/96

  Management of image or video recognition tasks · CPC title