Expression recognition method and related apparatus

Patent metadata

Field	Value
Publication number	US-12094247-B2
Application number	US-202117322710-A
Country	US
Kind code	B2
Filing date	May 17, 2021
Priority date	Mar 14, 2019
Publication date	Sep 17, 2024
Grant date	Sep 17, 2024

Abstract

An electronic device obtains an image that includes a face. The device performs feature extraction on the image, to obtain facial expression information corresponding to the face and facial feature information corresponding to the facial expression, wherein the facial feature information indicates an extent of the facial expression. The device determines facial emotion information according to the facial expression information. The device also determines facial feature expression information according to a target feature value corresponding to the facial emotion and the facial feature information. This expression recognition techniques disclosed herein can implement multi-task learning and reduce an amount of data required for model training, and can obtain both an emotion recognition result and a local expression recognition result, thereby improving efficiency and real-time performance of expression recognition and improving user experience.

First claim

What is claimed is:

1. An expression recognition method performed by an electronic device having one or more processors and memory, the method comprising:
obtaining an image that includes a face, wherein the face includes a plurality of facial action units, a displacement of each facial action unit from its default position indicating an extent of a facial expression associated with the facial action unit;
performing feature extraction on the image, to obtain global facial expression information corresponding to the face and local facial feature information corresponding to the face, wherein the local facial feature information indicates an extent of the displacement by a corresponding facial action unit in the face;
determining a global facial emotion probability value according to the global facial expression information;
determining a local facial feature expression probability value according to the local facial feature information of the corresponding facial action unit in the face; and
selecting, among a plurality of candidate facial emotions and a plurality of candidate facial expressions, a target facial emotion and a target facial expression corresponding to the target facial emotion for the face in the image according to the global facial emotion probability value and the local facial feature expression probability value.

2. The expression recognition method according to claim 1 , wherein performing feature extraction on the face image comprises:
inputting the image that includes the face to a key point learning model, the key point learning model comprising a backbone network and a key point recognition network connected to the backbone network;
performing feature extraction on the image through the backbone network, to obtain the global facial expression information; and
processing the global facial expression information through the key point recognition network, to obtain the local facial feature information.

3. The expression recognition method according to claim 2 , wherein performing feature extraction on the face image through the backbone network comprises:
performing convolution-activation-element addition-pooling operations on the image through the backbone network, to obtain the facial expression information.

4. The expression recognition method according to claim 2 , wherein the key point recognition network is a fully connected layer, and processing the facial expression information comprises:
integrating the facial expression information through the fully connected layer, to obtain the facial feature information.

5. The expression recognition method according to claim 1 , wherein determining a global facial emotion probability value according to the global facial expression information comprises:
integrating the global facial expression information corresponding to the face using a first expression recognition module; and
classifying the integrated global facial expression information, to obtain the global facial emotion probability value.

6. The expression recognition method according to claim 5 , wherein the first expression recognition module is a multi-layer perceptron model, and the multi-layer perceptron model comprises a first fully connected layer, a second fully connected layer, and an output layer that are sequentially connected.

7. The expression recognition method according to claim 6 , wherein
integrating the global facial expression information using the first expression recognition module comprises:
performing multi-dimensional mapping on the global facial expression information through the first fully connected layer, to output feature information; and
integrating the feature information through the second fully connected layer, to obtain the integrated global facial expression information; and
classifying the integrated facial expression information, to obtain the global facial emotion probability value comprises:
performing normalization processing on the integrated global facial expression information through the output layer, to obtain the global facial emotion probability value.

8. The expression recognition method according to claim 1 , wherein determining a local facial feature expression probability value comprises:
performing calculation on the local facial feature information using a second expression recognition module to obtain a target feature value, and determining the local facial feature expression probability value according to the target feature value.

9. The expression recognition method according to claim 8 , wherein the local facial feature information comprises key point coordinates and key point marks, corresponding to a facial action unit in the face; and
performing calculation on the local facial feature information comprises:
determining a target facial action unit in the face according to the key point marks;
determining feature points of the target facial action unit, and target key point coordinates corresponding to the feature points;
calculating the target feature value according to the target key point coordinates; and
obtaining the local facial feature expression probability value according to the target feature value, a first parameter, and a second parameter.

10. The expression recognition method according to claim 9 , wherein the target feature value comprises at least one of an aspect ratio, a slope, and a curvature of the target facial action unit in the face, and the first parameter is a preset slope, and the second parameter is a preset bias value.

11. The expression recognition method according to claim 9 , wherein obtaining the local facial feature expression probability value according to the target feature value, the first parameter, and the second parameter comprises:
determining a state value according to the target feature value, the first parameter, and the second parameter;
comparing the state value with the number “0”, to determine a maximum value; and
comparing the maximum value with the number “1”, to determine a minimum value, and using the minimum value as the expression probability value.

12. The expression recognition method according to claim 8 , wherein the local facial feature information comprises key point coordinates and key point marks, corresponding to a facial action unit in the face; and
performing calculation on the local facial feature information comprises:
determining a target facial action unit in the face image according to the key point marks;
matching key point coordinates corresponding to the target facial action unit against key point coordinates corresponding to the key point marks in a plurality of preset templates associated with the target facial action unit, to obtain a plurality of target feature values; and
performing normalization processing on the plurality of target feature values, to obtain the local facial feature expression probability value.

13. The expression recognition method according to claim 9 , further comprising prior to determining the target facial action unit in the face:
performing coordinate scale normalization processing on the key point coordinates.

14. An electronic device, comprising:
one or more processors; and
memory storing one or more programs, the one or more programs comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
obtaining an image that includes a face, wherein the face includes a plurality of facial action units, a displacement of each facial action unit from its default position indicating an extent of a facial expression associated with the fac

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Ji Xing
• Wang Yitong
• Zhou Zheng

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title