Method for detecting hand key points, method for recognizing gesture, and related devices

Patent metadata

Field	Value
Publication number	US-12299920-B2
Application number	US-202017780694-A
Country	US
Kind code	B2
Filing date	Aug 7, 2020
Priority date	Nov 29, 2019
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

Provided is a method for detecting hand key points. The method includes: acquiring a hand image to be detected; acquiring heat maps of the hand key points by inputting the hand image into a pre-trained heat map model, wherein the heat maps include two-dimensional coordinates of the hand key points; acquiring hand structured connection information by inputting the heat maps and the hand image into a pre-trained three-dimensional information prediction model; and determining, based on the hand structured connection information and the two-dimensional coordinates in the heat maps, three-dimensional coordinates of the hand key points in a world coordinate system.

First claim

What is claimed is:

1. A method for detecting hand key points, comprising:
acquiring a hand image to be detected;
acquiring heat maps of the hand key points by inputting the hand image into a pre-trained heat map model, wherein the heat maps comprise two-dimensional coordinates of the hand key points;
acquiring hand structured connection information by inputting the heat maps and the hand image into a pre-trained three-dimensional information prediction model;
determining, based on the hand structured connection information and the two-dimensional coordinates in the heat maps, three-dimensional coordinates of the hand key points in a world coordinate system; and
recognizing, based on three-dimensional coordinates of the hand key points in the world coordinate system, a gesture expressed by a hand in the hand image to detect the hand key points in real time through a mobile terminal;
wherein the hand structured connection information comprises joint bending angles formed by the hand key points and Euler angles of the hand; and determining, based on the hand structured connection information and the two-dimensional coordinates in the heat maps, the three-dimensional coordinates of the hand key points in the world coordinate system comprises:
calculating, based on the joint bending angles, a first direction vector of a vector formed by two hand key points in a hand coordinate system of the hand;
converting the first direction vector into a second direction vector in the world coordinate system based on the Euler angles;
calculating a vector length of the vector based on the two-dimensional coordinates in the heat maps;
acquiring the vector by calculating a product of the vector length and the second direction vector; and

calculating the three-dimensional coordinates of the two hand key points forming the vector in the world coordinate system based on the vector.

2. The method according to claim 1 , wherein acquiring the hand image to be detected comprises:
acquiring an original image;
detecting a hand from the original image; and
capturing an image having a preset size and containing the hand as the hand image to be detected.

3. The method according to claim 1 , wherein acquiring the heat maps of the hand key points by inputting the hand image into the pre-trained heat map model comprises:
acquiring the heat map of each hand key point by inputting the hand image into the pre-trained heat map model, wherein a size of the heat map of each hand key point is equal to a size of the hand image.

4. The method according to claim 1 , wherein the heat maps comprise heat maps of all hand key points, and acquiring the hand structured connection information by inputting the heat maps and the hand image into the pre-trained three-dimensional information prediction model comprises:
acquiring joint bending angles formed by the hand key points and Euler angles of the hand by inputting the heat maps of all hand key points and the hand image into the pre-trained three-dimensional information prediction model.

5. The method according to claim 1 , wherein calculating, based on the joint bending angles, the first direction vector of the vector formed by the two hand key points in the hand coordinate system of the hand comprises:
determining, based on a pre-established hand model, a first direction vector of a vector from a wrist key point to a proximal phalanx key point of a middle finger;
calculating a first direction vector of a vector from the wrist key point to a proximal phalanx key point of each finger based on the joint bending angles and the first direction vector of the vector from the wrist key point to the proximal phalanx key point of the middle finger; and
calculating the first direction vector of the vector between the two key points connected by the phalanxes of each finger based on the joint bending angles and the first direction vector of the vector from the wrist key point to the proximal phalanx key point of each finger.

6. The method according to claim 1 , wherein converting the first direction vector into the second direction vector in the world coordinate system based on the Euler angles comprises:
calculating an Euler rotation matrix based on the Euler angles; and
acquiring the second direction vector of the first direction vector in the world coordinate system by calculating a product of the first direction vector and the Euler rotation matrix.

7. The method according to claim 1 , wherein calculating the vector length of the vector based on the two-dimensional coordinates in the heat maps comprises:
determining the two hand key points forming the vector;
determining, based on the heat maps of the two hand key points, two-dimensional coordinates of the two hand key points; and
calculating the vector length of the vector based on the two-dimensional coordinates of the two hand key points.

8. The method according to claim 7 , wherein
each pixel point on the heat maps is associated with a probability value, wherein the probability value represents a probability of the hand key point at each pixel point; and
determining, based on the heat maps of the two hand key points, the two-dimensional coordinates of the two hand key points comprises:
determining a pixel point with a greatest probability value from the heat map of each hand key point;
acquiring local two-dimensional coordinates by acquiring coordinates of the pixel point with the greatest probability value in the heat map; and
acquiring the two-dimensional coordinates of each hand key point by converting the local two-dimensional coordinates to coordinates in the hand image.

9. The method according to claim 1 , wherein calculating the three-dimensional coordinates of the two hand key points forming the vector in the world coordinate system based on the vector comprises:
acquiring the three-dimensional coordinates of a wrist key point in the hand key points in the world coordinate system; and
calculating the three-dimensional coordinates of the two hand key points forming the vector in the world coordinate system based on the three-dimensional coordinates of the wrist key point in the world coordinate system and the vector.

10. A method for recognizing a gesture, comprising:
acquiring a hand image to be recognized;
detecting key points in the hand image; and
recognizing, based on the key points, the gesture expressed by a hand in the hand image;
wherein detecting the key points in the hand image comprises: detecting the key points in the hand image according to a method for detecting hand key points, comprising:
acquiring a hand image to be detected;
acquiring heat maps of the hand key points by inputting the hand image into a pre-trained heat map model, wherein the heat maps comprise two-dimensional coordinates of the hand key points;
acquiring hand structured connection information by inputting the heat maps and the hand image into a pre-trained three-dimensional information prediction model;
determining, based on the hand structured connection information and the two-dimensional coordinates in the heat maps, three-dimensional coordinates of the hand key points in a world coordinate system; and
recognizing, based on three-dimensional coordinates of the hand key points in the world coordinate system, the gesture expressed by the hand in the hand image to detect the hand key points in real time through a mobile terminal;
wherein the hand structured connection information comprises joint bending angles formed by the hand key points and Euler angles of the hand; and determining, based on the hand structured connection information and the two-dimensional coordinates in the heat maps, the three-dimensional coordinat

Assignees

• Bigo Tech Pte Ltd

Inventors

• Xiang Wei
• Wang Yifeng

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06V40/11

  Hand-related biometrics; Hand pose recognition · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title