Method and apparatus for training pose recognition model, and method and apparatus for image recognition

Patent metadata

Field	Value
Publication number	US-11907848-B2
Application number	US-202117330261-A
Country	US
Kind code	B2
Filing date	May 25, 2021
Priority date	Apr 12, 2019
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

This application provides a method for training a pose recognition model performed at a computer device. The method includes: inputting a sample image labeled with human body key points into a feature map model included in a pose recognition model, to output a feature map of the sample image; inputting the feature map into a two-dimensional (2D) model included in the pose recognition model, to output 2D key point parameters used for representing a 2D human body pose; input a target human body feature map cropped from the feature map and the 2D key point parameter into a three-dimensional (3D) model included in the pose recognition model, to output 3D pose parameters used for representing a 3D human body pose; constructing a target loss function based on the 2D key point parameters and the 3D pose parameters; and updating the pose recognition model based on the target loss function.

First claim

What is claimed is:

1. A method for training a pose recognition model, applicable to a computer device, the method comprising:
processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image;
processing the feature map of the sample image by using a two-dimensional (2D) model comprised in the pose recognition model, to obtain 2D key point parameters used for representing a 2D human body pose;
processing, by using a three-dimensional (3D) model comprised in the pose recognition model, a target human body feature map cropped from the feature map and the 2D key point parameters, to obtain 3D pose parameters used for representing a 3D human body pose;
constructing a target loss function based on the 2D key point parameters and the 3D pose parameters; and
updating the pose recognition model based on the target loss function.

2. The method according to claim 1 , wherein before the processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image, the method further comprises:
determining the human body key points in a key point set according to a setting; and
labeling the sample image based on the human body key points and with reference to the key point set.

3. The method according to claim 2 , wherein the key point set comprises:
reference key points used for localization of human body parts and extension key points, which, together with the reference key points, represent a plurality of 3D poses of the parts to which the reference key points and the extension key points belong.

4. The method according to claim 1 , wherein the target loss function comprises a first loss function corresponding to the 3D model; and the constructing a target loss function based on the 2D key point parameters and the 3D pose parameters comprises:
determining corresponding 2D key point information based on the 3D pose parameters; and
constructing the first loss function based on the 2D key point parameters and the 2D key point information.

5. The method according to claim 4 , wherein the target loss function further comprises a loss function corresponding to the 2D model and a second loss function corresponding to the 3D model; the 2D key point parameters comprise a part affinity field (PAF) parameter of the human body key points and a heatmap of the human body key points, and the 3D pose parameters comprise a shape parameter and a pose parameter of a human body; and
the constructing a target loss function based on the 2D key point parameters and the 3D pose parameters comprises:
constructing the loss function corresponding to the 2D model based on a difference between a PAF parameter outputted by the 2D model and the PAF parameter of the human body key points in the sample image and a difference between a heatmap outputted by the 2D model and the heatmap of the corresponding human body key points in the sample image; and
constructing the second loss function corresponding to the 3D model based on a difference between a shape parameter outputted by the 3D model and the shape parameter of the corresponding human body in the sample image and a difference between a pose parameter outputted by the 3D model and the pose parameter of the corresponding human body in the sample image.

6. The method according to claim 1 , wherein before the processing a target human body feature map cropped from the feature map and the 2D key point parameters by using a 3D model comprised in the pose recognition model, to obtain a 3D pose parameter used for representing a 3D human body pose, the method further comprises:
determining a target human body in the feature map based on the 2D key point parameters; and
cropping the feature map according to the target human body, to obtain the target human body feature map.

7. The method according to claim 1 , wherein the updating a model parameter of the pose recognition model based on the target loss function comprises:
determining a value of the target loss function based on the 2D key point parameters and the 3D pose parameters;
determining an error signal of the pose recognition model based on the target loss function in a case that the value of the target loss function exceeds a preset threshold; and
back-propagating the error signal in the pose recognition model, and updating a model parameter of each layer during the propagation.

8. The method according to claim 1 , the method comprising:
processing, by using the feature map model comprised in the pose recognition model, a to-be-recognized image comprising a human body, to obtain a feature map of the to-be-recognized image;
processing the feature map of the to-be-recognized image by using the 2D model comprised in the pose recognition model, to obtain 2D key point parameters used for representing a 2D human body pose in the to-be-recognized image; and
processing, by using the 3D model comprised in the pose recognition model, a target human body feature map cropped from the feature map and the 2D key point parameters, to obtain 3D pose parameters used for representing a 3D human body pose in the to-be-recognized image.

9. The method according to claim 8 , further comprising:
recognizing a 2D pose of the human body in the to-be-recognized image based on the 2D key point parameter, the to-be-recognized image being acquired based on an outputted image of a specific human pose;
performing similarity matching between the 2D pose and the specific human pose, to obtain a matching result; and
outputting prompt information used for representing the matching result.

10. The method according to claim 8 , further comprising:
constructing, based on the 3D pose parameter, a 3D human body model corresponding to the target human body; and
controlling the 3D human body model to perform a target action, the target action matching an action performed by the target human body.

11. A computer device, comprising:
a memory, configured to store executable instructions; and
a processor, configured to perform, when executing the executable instructions stored in the memory, a plurality of operations including:
processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image;
processing the feature map of the sample image by using a two-dimensional (2D) model comprised in the pose recognition model, to obtain 2D key point parameters used for representing a 2D human body pose;
processing, by using a three-dimensional (3D) model comprised in the pose recognition model, a target human body feature map cropped from the feature map and the 2D key point parameters, to obtain 3D pose parameters used for representing a 3D human body pose;
constructing a target loss function based on the 2D key point parameters and the 3D pose parameters; and
updating the pose recognition model based on the target loss function.

12. The computer device according to claim 11 , wherein the plurality of operations further comprise:
before processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image:
determining the human body key points in a key point set according to a setting; and
labeling the sample image based on the human body key points and with reference to the key point set.

13. The computer device according to claim 12 , wherein the key point set comprises:
reference

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Luo Jingmin
• Zhu Xiaolong
• Wang Yitong
• Ji Xing

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title