Human posture detection utilizing posture reference maps

Patent metadata

Field	Value
Publication number	US-11908244-B2
Application number	US-201917297882-A
Country	US
Kind code	B2
Filing date	Nov 20, 2019
Priority date	Nov 27, 2018
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

Provided is a human posture detection method including: acquiring a plurality of frames of image data; acquiring a plurality of human posture reference maps output by a human posture detection model responsive to inputting a current frame of image data to the human posture detection model with reference to human posture confidence maps of a previous frame of image data, wherein different human posture reference maps correspond different human-posture key points; identifying a human-posture key point in each of the human posture reference maps; and generating human posture confidence maps of the current frame of image data based on credibility of the human-posture key points, wherein the human posture confidence maps of the current frame of image data is configured to participate in generation of human posture confidence maps of a next frame of image data.

First claim

What is claimed is:

1. A human posture detection method, comprising:
acquiring a plurality of frames of image data;
acquiring a plurality of human posture reference maps output by a human posture detection model responsive to inputting a current frame of image data to the human posture detection model with reference to human posture confidence maps of a previous frame of image data, wherein different human posture reference maps correspond different human-posture key points;
identifying a human-posture key point in each of the human posture reference maps; and
generating human posture confidence maps of the current frame of image data based on credibility of the human-posture key points, wherein the human posture confidence maps of the current frame of image data is configured to participate in generation of human posture confidence maps of a next frame of image data,
wherein the human posture detection model comprises a main path, a first branch, and a second branch, the main path comprising a residual module and an up-sampling module, the first branch comprising a refinement network module, and the second branch comprising a feedback module; and
acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting the current frame of image data into the human posture detection model with reference to the human posture confidence maps of the previous frame of image data comprises:
acquiring a first convolution result output by the residual module based on a processing result of the residual module on the current frame of image data and a processing result of the feedback module on the human posture confidence maps of the previous image data;
acquiring a second convolution result by inputting the first convolution result output by the residual module into the up-sampling module and processing the first convolution result therein, and acquiring a third convolution result by inputting the first convolution result output by the residual module into the refinement network module and processing the first convolution result therein; and
acquiring the plurality of human posture reference maps by adding up the second convolution result and the third convolution result.

2. The method according to claim 1 , wherein acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting the current frame of image data to the human posture detection model with reference to the human posture confidence maps of the previous frame of image data comprises:
acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting, in response to the human posture confidence maps of the previous frame of image data being credible, the current frame of image data and the human posture confidence maps of the previous frame of image data into the human posture detection model; and
acquiring the plurality of human posture reference maps output by the human posture detection model responsive by inputting, in response to the human posture confidence maps of the previous frame of image data being incredible, the current frame of image data and target image data into the human posture detection model, wherein the target image data is image data containing no prior knowledge.

3. The method according to claim 1 , wherein each of the human posture reference maps comprises a plurality of candidate points for the human-posture key point, and a coordinate position of each of the candidate points corresponds to one probability value; and
identifying the human-posture key points in each of the human posture reference maps comprises:
determining a coordinate position corresponding to a maximum probability value among a plurality of probability values, and taking a candidate point corresponding to the coordinate position as the human-posture key point.

4. The method according to claim 2 , wherein generating the human posture confidence maps of the current frame of image data based on the credibility of the human-posture key points comprises:
generating, in response to the human-posture key points being credible, mask patterns with the human-posture key points as centers, and taking the mask patterns as the human posture confidence maps; and
taking, in response to the human-posture key points being incredible, the target image data as the human posture confidence maps.

5. The method according to claim 4 , further comprising:
determining, in response to probability values corresponding to the human-posture key points being greater than a preset threshold value, that the human-posture key points are credible; and
determining, in response to the probability values corresponding to the human-posture key points being less than or equal to the preset threshold value, that the human-posture key points are incredible.

6. The method according to claim 1 , wherein the residual module comprises a first residual unit, a second residual unit, and a third residual unit; and
acquiring the first convolution result output by the residual module based on a processing result of the residual module on the current frame of image data and a processing result of the feedback module on the human posture confidence maps of the previous image data comprises:
acquiring a first intermediate result by inputting the current frame image data into the first residual unit and processing the current frame image data therein;
inputting the human posture confidence maps of the previous image data into the feedback module and processing the human posture confidence maps of the previous image data therein;
acquiring an addition result by adding up the first intermediate result and a processing result output by the feedback module;
acquiring a second intermediate result by inputting the addition result into the second residual unit and processing the addition result therein; and
acquiring a third intermediate result by inputting the second intermediate result into the third residual unit and processing the second intermediate result therein, and taking the third intermediate result as the first convolution result;
wherein numbers of channels for the first intermediate result, the second intermediate result, and the third intermediate result are increased successively.

7. The method according to claim 6 , wherein the human posture detection model further comprises a third branch; and
acquiring the second convolution result by inputting the first convolution result output by the residual module into the up-sampling module and processing the first convolution result therein comprise:
acquiring a fourth intermediate result by inputting the first intermediate result into the third branch and processing the first intermediate result therein;
acquiring a fifth intermediate result by inputting the second intermediate result into the third branch and processing the second intermediate result therein;
acquiring a sixth intermediate result by inputting the third intermediate result and the fifth intermediate result into the up-sampling module and processing the third intermediate result and the fifth intermediate result therein; and
acquiring a seventh intermediate result by inputting the fourth intermediate result and the sixth intermediate result into the up-sampling module and processing the fourth intermediate result and the sixth intermediate result therein, and taking the seventh intermediate result as the second convolution result;
wherein numbers of channels for the sixth intermediate result and the seventh intermediate result are decreased successively.

8. The method according to claim 1 , acquiring the plurality of human posture reference maps by

Assignees

• Bigo Tech Pte Ltd

Inventors

• Xiang Wei
• Wang Yifeng
• Huang Qiushi
• Liang Zhujin

Classifications

• G06V40/23 Primary

  Recognition of whole body movements, e.g. for sport training · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T7/251

  involving models · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title