Method and apparatus with 3D modeling of human body

Patent metadata

Field	Value
Publication number	US-12067679-B2
Application number	US-202217732803-A
Country	US
Kind code	B2
Filing date	Apr 29, 2022
Priority date	Nov 29, 2021
Publication date	Aug 20, 2024
Grant date	Aug 20, 2024

Abstract

A method with three-dimensional (3D) modeling of a wearer of a wearable device includes generating a feature map for each of a plurality of images of the wearer obtained from a plurality of imaging devices provided in the wearable device, obtaining joint keypoint information corresponding to joint positions of the wearer and initial shape coefficient information associated with a shape of the wearer based on the feature map for each of the images, determining a target 3D joint angle for 3D modeling of the wearer based on the joint keypoint information and the initial shape coefficient information, determining target shape coefficient information for 3D modeling of the wearer based on the joint keypoint information and the initial shape coefficient information, and obtaining a 3D mesh of the wearer based on the target 3D joint angle and the target shape coefficient information.

First claim

What is claimed is:

1. A processor-implemented method with three-dimensional (3D) modeling of a wearer of a wearable device, comprising:
generating a feature map for each of a plurality of images of the wearer obtained from a plurality of imaging devices provided in the wearable device;
obtaining joint keypoint information corresponding to joint positions of the wearer and initial shape coefficient information associated with a shape of the wearer based on the feature map, for each of the images, wherein the joint keypoint information comprises two-dimensional (2D) pixel information and depth information;
determining a target 3D joint angle for 3D modeling of the wearer based on the joint keypoint information and the initial shape coefficient information;
determining target shape coefficient information for 3D modeling of the wearer based on the joint keypoint information and the initial shape coefficient information; and
obtaining a 3D mesh of the wearer based on the target 3D joint angle and the target shape coefficient information,
wherein the obtaining of the initial shape coefficient information includes obtaining the initial shape coefficient through a performed regression only on the feature map of a first frame.

2. The processor-implemented method of claim 1 , wherein the feature map is generated based on at least one of a deep convolutional neural network, a deep convolutional neural network with depthwise separable convolutions, and a deep residual learning neural network.

3. The processor-implemented method of claim 1 , wherein the obtaining of the joint keypoint information and the initial shape coefficient information for each of the images comprises:
obtaining the 2D pixel information inferred based on the feature map and a first convolutional neural network (CNN) model; and
obtaining the depth information inferred based on the feature map and a second CNN model.

4. The processor-implemented method of claim 1 , wherein the determining of the target 3D joint angle comprises:
calculating an error of the 2D pixel information;
calculating an error of the depth information;
calculating a 3D joint angle error with respect to time;
calculating a total error of a 3D joint angle based on the error of the 2D pixel information, the error of the depth information, and the 3D joint angle error with respect to time; and
determining, to be the target 3D joint angle, a 3D joint angle minimizing the total error.

5. The processor-implemented method of claim 4 , wherein the 3D joint angle error with respect to time is calculated based on Equation 4,
E temp =∥θ−θ t-1 ∥ 2 2   Equation 4:

wherein E temp denotes the 3D joint angle error with respect to time, θ denotes a 3D joint angle, θ t-1 denotes a 3D joint angle in an immediately previous frame, and ∥ ∥ 2 2 is a square of an L2-norm that denotes a sum of squares of respective components.

6. The processor-implemented method of claim 4 , wherein the imaging devices comprise a left imaging device provided on a left side of the wearable device and a right imaging device provided on a right side of the wearable device, and the error of the 2D pixel information is calculated based on Equation 1,

E

2
⁢
D

=

∑
i



v

l
⁢
i

(

∏

l

⁢

(

X
i

(

θ
,

β

t
-
1

)

)

-

p
^

li

)



2
2

+



v
ri

(

∏

r

⁢

(

X
i

(

θ
,

β

t
-
1

Assignees

• Samsung Electronics Co Ltd

Inventors

• Jee Seunghoon

Classifications

• G06T7/50

  Depth or shape recovery · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T2207/20044

  Skeletonization; Medial axis transform · CPC title