3D hand shape and pose estimation

Patent metadata

Field	Value
Publication number	US-11734844-B2
Application number	US-202217823764-A
Country	US
Kind code	B2
Filing date	Aug 31, 2022
Priority date	Dec 5, 2018
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for receiving a monocular image that includes a depiction of a hand and extracting features of the monocular image using a plurality of machine learning techniques. The program and method further include modeling, based on the extracted features, a pose of the hand depicted in the monocular image by adjusting skeletal joint positions of a three-dimensional (3D) hand mesh using a trained graph convolutional neural network (CNN); modeling, based on the extracted features, a shape of the hand in the monocular image by adjusting blend shape values of the 3D hand mesh representing surface features of the hand depicted in the monocular image using the trained graph CNN; and generating, for display, the 3D hand mesh adjusted to model the pose and shape of the hand depicted in the monocular image.

First claim

What is claimed is:

1. A method comprising:
obtaining a first plurality of images that include respective representations of a hand, one or more of the representations of the hand comprising a synthetic representation of the hand;
obtaining a second plurality of images that include real-world depictions of a hand and reference three-dimensional (3D) depth maps;
generating a pseudo-ground truth mesh of each of the real-world depictions of the hand using a trained graph convolutional neural network (CNN);
training a first machine learning technique based on a first feature of the first plurality of images;
training a second machine learning technique based on a second feature of the first plurality of images separately from the first machine learning technique; and
generating a 3D hand mesh by applying the trained first and second machine learning techniques and the trained graph CNN to one or more monocular images.

2. The method of claim 1 , wherein the synthetic representation of the hand comprises a graphical representation of the hand further, further comprising:
training the first and second machine learning techniques together with the graph CNN based on the first plurality of images.

3. The method of claim 2 , further comprising:
based on the first and second machine learning techniques, continuously changing an appearance of a 3D hand mesh by continuously capturing new monocular images of the hand in different positions, wherein the appearance of the 3D hand mesh changes to resemble the different positions of the hand as the hand changes from one position to another position.

4. The method of claim 1 , further comprising training the first and second machine learning techniques together with the trained graph CNN based on the pseudo-ground truth mesh of each of the real-world depictions of the hand, the second plurality of images of the real-world depictions of a hand and reference 3D depth maps.

5. The method of claim 1 , further comprising:
receiving a given monocular image that includes a depiction of a hand; and
modeling a pose of the hand depicted in the given monocular image by adjusting skeletal joint positions of a 3D hand mesh by estimating 3D coordinates of vertices in the 3D hand mesh using the trained graph CNN.

6. The method of claim 5 , further comprising:
linearly regressing the joint positions using a linear graph CNN; and
generating, for display, the 3D hand mesh adjusted to model the pose of the hand depicted in the given monocular image.

7. The method of claim 1 , further comprising:
applying the first machine learning technique to a given monocular image to estimate a two-dimensional (2D) heat map of the hand in the given monocular image and to generate an image feature map; and
encoding a 2D heat map and the image feature map using the second machine learning technique to generate a feature vector.

8. The method of claim 1 , wherein the first machine learning technique comprises a stacked hourglass network, and wherein the second machine learning technique comprises a residual network.

9. The method of claim 1 further comprising:
modeling based on one or more extracted features of a given monocular image, a shape of the hand in the given monocular image by adjusting blend shape values of a 3D hand mesh representing surface features of the hand depicted in the given monocular image using the trained graph CNN.

10. The method of claim 1 , further comprising generating an image of the first plurality of images by:
generating a 3D hand model by combining a plurality of hand joints with a plurality of surface textures; and
combining the generated hand model with a background image.

11. The method of claim 10 , further comprising:
randomly selecting a hand pose from a plurality of hand poses;
adjusting the plurality of hand joints based on the selected hand pose; and
adjusting the plurality of surface textures by applying random weights to blend shapes and ratios.

12. The method of claim 10 , wherein generating the 3D hand model comprises:
obtaining a 3D hand model that includes a first level of coarseness having a first number of vertices;
applying the trained graph CNN to the first level of coarseness;
upsampling the obtained 3D hand model to increase the level of coarseness to a second level of coarseness having a second number of vertices greater than the first number of vertices;
generating a tree structure representing correspondences of vertices in the first and second levels of coarseness; and
updating the trained graph CNN based on the upsampled 3D hand model and the generated tree structure.

13. The method of claim 1 , further comprising:
training the first machine learning technique based on a heat map loss function and training the second machine learning technique based on a 3D pose loss function, and the first and second machine learning techniques are trained together with the graph CNN by training the first and second machine learning techniques together based on the heat map loss function, the 3D pose loss function, and a mesh loss function.

14. The method of claim 1 , further comprising:
training the first and second machine learning techniques and the trained graph CNN based on the generated pseudo-ground truth mesh, the real-world depictions of the hand, and the reference 3D depth maps of the real-world depictions of the hand.

15. The method of claim 1 , further comprising:
modeling, based on one or more extracted features of a given monocular image, a shape of the hand in the given monocular image by adjusting blend shape values of a 3D hand mesh representing surface features of the hand depicted in the given monocular image using the trained graph CNN.

16. A system comprising:
a processor configured to perform operations comprising:
obtaining a first plurality of images that include respective representations of a hand, one or more of the representations of the hand comprising a synthetic representation of the hand;
obtaining a second plurality of images that include real-world depictions of a hand and reference three-dimensional (3D) depth maps;
generating a pseudo-ground truth mesh of each of the real-world depictions of the hand using a trained graph convolutional neural network (CNN);
training a first machine learning technique based on a first feature of the first plurality of images;
training a second machine learning technique based on a second feature of the first plurality of images separately from the first machine learning technique; and
generating a 3D hand mesh by applying the trained first and second machine learning techniques and the trained graph CNN to one or more monocular images.

17. The system of claim 16 , wherein the operations further comprise:
modeling, based on one or more extracted features of a given monocular image, a shape of the hand in the given monocular image by adjusting blend shape values of a 3D hand mesh representing surface features of the hand depicted in the given monocular image using the trained graph CNN.

18. The system of claim 16 , wherein the first machine learning technique comprises a stacked hourglass network, and wherein the second machine learning technique comprises a residual network.

19. A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
obtaining a first plurality of images that include respective representations of a hand, one or more of the repr

Assignees

• Snap Inc

Inventors

• Ge Liuhao
• Ren Zhou
• Li Yuncheng
• Xue Zehao
• Wang Yingying

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T7/50 Primary

  Depth or shape recovery · CPC title

• G06N3/045

  Combinations of networks · CPC title