Method and system for constructing personalized avatars using a parameterized deformable mesh

Patent metadata

Field	Value
Publication number	US-9524582-B2
Application number	US-201514604829-A
Country	US
Kind code	B2
Filing date	Jan 26, 2015
Priority date	Jan 28, 2014
Publication date	Dec 20, 2016
Grant date	Dec 20, 2016

Abstract

A method and apparatus for generating a 3D personalized mesh of a person from a depth camera image for medical imaging scan planning is disclosed. A depth camera image of a subject is converted to a 3D point cloud. A plurality of anatomical landmarks are detected in the 3D point cloud. A 3D avatar mesh is initialized by aligning a template mesh to the 3D point cloud based on the detected anatomical landmarks. A personalized 3D avatar mesh of the subject is generated by optimizing the 3D avatar mesh using a trained parametric deformable model (PDM). The optimization is subject to constraints that take into account clothing worn by the subject and the presence of a table on which the subject in lying.

First claim

What is claimed is:

1. A method of generating a personalized 3D avatar mesh of a human subject from a depth camera image for medical imaging scan planning, comprising:
converting a depth camera image of a subject to a 3D point cloud;
detecting a plurality of anatomical landmarks in the 3D point cloud;
initializing a 3D avatar mesh by aligning a template mesh to the 3D point cloud based on the detected anatomical landmarks; and
generating a personalized 3D avatar mesh of the subject by optimizing the 3D avatar mesh using a trained parametric deformable model (PDM)

wherein generating a personalized 3D avatar mesh of the subject by optimizing the 3D avatar mesh using a trained parametric deformable model (PDM) comprises:
optimizing parameters of the trained PDM that control a pose and shape of the 3D avatar mesh and locations of vertices of the 3D avatar mesh to minimize a cost function, and wherein the cost function includes a table constraint that causes the optimization to move vertices of the 3D avatar above a table on which the subject is lying.

2. The method of claim 1 , wherein the depth camera image comprises an red, green, blue (RGB) image and a depth image, and converting a depth camera image of a subject to a 3D point cloud comprises:
for each of a plurality of pixels in the RGB image, mapping that pixel to a location in the 3D point cloud based on a corresponding depth value in the depth image.

3. The method of claim 1 , wherein detecting a plurality of anatomical landmarks in the 3D point cloud comprises:
detecting a plurality of joint landmarks corresponding to locations of the subject's joints in the 3D point cloud.

4. The method of claim 1 , wherein detecting a plurality of anatomical landmarks in the 3D point cloud comprises:
detecting each of the plurality of anatomical landmarks using a respective trained classifier.

5. The method of claim 1 , wherein the trained PDM comprises a trained pose deformation model and a trained shape deformation model.

6. The method of claim 5 , wherein the trained PDM is trained based on a plurality of training instances and the plurality of training instances comprises a plurality of synthetic human meshes having a plurality of poses and body shapes generated using a 3D rendering software.

7. The method of claim 1 , wherein the cost function includes a clothing constraint that causes the optimization to move vertices of the 3D avatar in a clothing region below corresponding points of the 3D point cloud.

8. The method of claim 7 , wherein the clothing constraint is based on a probability calculated for each vertex of the 3D avatar mesh that the vertex in is a non-clothing region and the probability for each vertex is calculated using a trained probability model.

9. The method of claim 1 , wherein the cost function includes smoothness constraint that penalizes deformations of a vertex in the 3D avatar mesh that are different from an average deformation of neighboring vertices to that vertex.

10. The method of claim 1 , wherein the cost function to be minimized is:
Σ k Σ j=2,3 ∥R k F U,μ Γ a k {circumflex over (v)} k,j −( y j,k −y 1,k )∥ 2 +d cloth ( Y )+ d table ( Y )+ s ( Y ),

where R k is the rigid rotation matrix, F U,μ is the trained shape deformation model, Γ a k is the trained pose deformation model, {circumflex over (v)} k,j denotes edges of a triangle in the template mesh, y denotes vertices of the estimated avatar mesh model, L is the set of correspondences between the avatar vertex y l and the corresponding point z l in the 3D point cloud, d cloth (Y) is a clothing constraint, d table (Y) is a table constraint, and s(Y) is a smoothing constraint.

11. The method of claim 10 , wherein d cloth (Y)=Σ lεL ∥ε l +n z l T (z l −y l )∥ 2 , where n z l is the normal of the point z l , a vertex of the avatar mesh is below the 3D point cloud when n z l T (z l −y l )>0, ∀lεL, ε l is determined by e k =(1−P(y k ))τ, where kεX, P(y k ) represents a probability, calculated using a trained probability model, that a vertex y k of the 3D avatar mesh belongs to a non-clothing region, and τ is a threshold that represents the maximal distance between the avatar surface and the clothing surface, including the thickness of the clothing.

12. The method of claim 10 , wherein d table (Y)=Σ kεX,y k z <h table ∥ε table +y k z −h table ∥ 2 , where y k z is a z-axis value of the avatar vertex y k , h table is the current table height, and e table ensures that the avatar mesh lies sufficiently above the table.

13. The method of claim 10 , wherein

s
⁡

(
y
)

=

∑

k
∈
X

⁢



(

y
k

-

y
^

k

)

-

1

[

N
k

]

⁢

∑

j
∈

N
k

⁢

(

y
j

-

y
^

j

)



2

,

where y denotes the vertices of a new avatar mesh, ŷ denotes the vertices of a current avatar mesh, and N k is the set o

Assignees

• Siemens Ag
• Siemens Healthcare Gmbh

Inventors

• Ma Kai
• Chen Terrence
• Singh Vivek Kumar
• Chang Yao-Jen
• Wels Michael
• Soza Grzegorz

Classifications

• G06T2207/10012

  Stereo images · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title