Action recognition and pose estimation method and apparatus

Patent metadata

Field	Value
Publication number	US-11478169-B2
Application number	US-202016846890-A
Country	US
Kind code	B2
Filing date	Apr 13, 2020
Priority date	Oct 13, 2017
Publication date	Oct 25, 2022
Grant date	Oct 25, 2022

Abstract

Action recognition methods are disclosed. An embodiment of the methods includes: identifying a video that comprises images of a human body to be processed; identifying at least one image to be processed, wherein the at least one image is at least one of an optical flow image generated based on a plurality of frames of images in the video, or a composite image of one or more frames of images in the video; performing convolution on the at least one image to obtain a plurality of eigenvectors, wherein the plurality of eigenvectors indicate a plurality of features of different locations in the at least one image; determining a weight coefficient set of each of a plurality of human joints of the human body based on the plurality of eigenvectors, wherein the weight coefficient set comprises a weight coefficient of each of the plurality of eigenvectors for the human joint; weighting the plurality of eigenvectors based on the weight coefficient set to obtain an action feature of each of the plurality of human joints; determining an action feature of the human body based on the action feature of each of the human joints; and determining an action type of the human body based on the action feature of the human body.

First claim

What is claimed is:

1. An action recognition method, comprising:
identifying a video to be processed that comprises images of a human body;
identifying at least one image to be processed, wherein the at least one image is at least one of an optical flow image generated based on a plurality of frames of images in the video, or a composite image of one or more frames of images in the video;
performing convolution on the at least one image to obtain a plurality of eigenvectors, wherein the plurality of eigenvectors indicate a plurality of features of different locations in the at least one image;
determining a weight coefficient set of each of a plurality of human joints of the human body based on the plurality of eigenvectors, wherein the weight coefficient set comprises a weight coefficient of each of the plurality of eigenvectors for the human joint a distance between a corresponding location of the first-type eigenvector in the at least one image and a corresponding location of the human joint in the at least one image is a first distance, a distance between a corresponding location of the second-type eigenvector in the at least one image and the corresponding location of the human joint in the at least one image is a second distance, the first distance is less than or equal to the second distance;
weighting the plurality of eigenvectors based on the weight coefficient set to obtain an action feature of each of a plurality of human joints;
determining an action feature of the human body based on the action feature of each of the human joints; and
determining an action type of the human body based on the action feature of the human body.

2. The method according to claim 1 , wherein
the weight coefficient set of each of the human joints is further determined based on cumulative action information of the human body, wherein the cumulative action information of the human body indicates an action feature of the human body accumulated before a current time point.

3. The method according to claim 1 , wherein the action feature of the human body is determined by
weighting or combining the action feature of the plurality of human joints.

4. The method according to claim 1 , wherein the action feature of the human body is determined by weighting or combining action feature of each of a plurality of human parts of the human body, and wherein the action feature of each of the plurality of human parts is determined by weighting or combining the action feature of each of the plurality of human joints.

5. The method according to claim 1 , wherein the at least one image comprises the composite image and the optical flow image, and the performing convolution on the at least one image to obtain a plurality of eigenvectors comprises:
performing convolution on the composite image to obtain a plurality of eigenvectors of the composite image; and
performing convolution on the optical flow image to obtain a plurality of eigenvectors of the optical flow image;
the determining the weight coefficient set comprises:
determining a first-type weight coefficient set of each of the plurality of human joints based on the plurality of eigenvectors of the composite image; and
determining a second-type weight coefficient set of each of the plurality of human joints based on the plurality of eigenvectors of the optical flow image;
the weighting the plurality of eigenvectors based on the weight coefficient set comprises:
weighting the plurality of eigenvectors of the composite image based on the first-type weight coefficient set to obtain a first-type action feature of each of the plurality of human joints; and
weighting the plurality of eigenvectors of the optical flow image based on the second-type weight coefficient set to obtain a second-type action feature of each of the plurality of human joints; and wherein
the action feature of the human body is determined by
weighting or combining first-type action features of the plurality of human joints and second-type action features of the plurality of human joints.

6. A pose estimation method, comprising:
identifying a video to be processed that comprises images of a human body;
identifying at least one image to be processed, wherein the at least one image is at least one of an optical flow image generated based on a plurality of frames of images in the video, or a composite image of one or more frames of images in the video;
performing convolution on the at least one image to obtain a plurality of eigenvectors, wherein the plurality of eigenvectors indicate a plurality of features of different locations in the at least one image;
determining a weight coefficient set of each of a plurality of human joints of the human body based on the plurality of eigenvectors, wherein the weight coefficient set comprises a weight coefficient of each of the plurality of eigenvectors for the human joint, a weight coefficient of a first-type eigenvector is greater than or equal to a weight coefficient of a second-type eigenvector, a distance between a corresponding location of the first-type eigenvector in the at least one image and a corresponding location of the human joint in the at least one image is a first distance, a distance between a corresponding location of the second-type eigenvector in the at least one image and the corresponding location of the human joint in the at least one image is a second distance, the first distance is less than or equal to the second distance;
determining a corresponding region of a first eigenvector of the plurality of eigenvectors in the at least one image as a location of the human joint, wherein a weight coefficient of the first eigenvector is a first weight coefficient, and the first weight coefficient is a weight coefficient greater than a preset threshold in the weight coefficient set of the human joint; and
determining a pose of the human body based on the location of the human joint.

7. The method according to claim 6 , wherein
the weight coefficient set of each of the human joints is further determined based on cumulative action information of the human body, wherein the cumulative action information of the human body indicates an action feature of the human body accumulated before a current time point.

8. The method according to claim 6 , wherein the at least one image comprises the composite image and the optical flow image, and the performing convolution on the at least one image to obtain a plurality of eigenvectors comprises:
performing convolution on the composite image to obtain a plurality of eigenvectors of the composite image; and
performing convolution on the optical flow image to obtain a plurality of eigenvectors of the optical flow image;
the determining the weight coefficient set comprises:
determining a first-type weight coefficient set of each of the plurality of human joints based on the plurality of eigenvectors of the composite image; and
determining a second-type weight coefficient set of each of the plurality of human joints based on the plurality of eigenvectors of the optical flow image; and
determining the weight coefficient set of each human joint based on the first-type weight coefficient set and the second-type weight coefficient set.

9. The method according to claim 8 , wherein the weight coefficient set of each human joint is determined based on weighting the first-type weight coefficient set and the second-type weight coefficient set.

10. An action recognition device, comprising at least one processor and a non-transitory medium storing program instructions, wherein the at least one processor, by executing the program instructions, causes the action recognition device to:
identifying a video to be processed

Assignees

• Huawei Tech Co Ltd

Inventors

• Qiao Yu
• Du Wenbin
• Wang Yali
• Jiang Lihui
• Liu Jianzhuang

Classifications

• G06V40/20 Primary

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• A61B5/1128 Primary

  using image analysis (A61B5/1127 takes precedence) · CPC title

• G06N3/08

  Learning methods · CPC title

• A61B5/1116

  Determining posture transitions · CPC title