3D body modeling from one or more depth cameras in the presence of articulated motion

Patent metadata

Field	Value
Publication number	US-9418475-B2
Application number	US-201313801099-A
Country	US
Kind code	B2
Filing date	Mar 13, 2013
Priority date	Apr 25, 2012
Publication date	Aug 16, 2016
Grant date	Aug 16, 2016

Abstract

The present disclosure describes systems and techniques relating to generating three dimensional (3D) models from range sensor data. According to an aspect, multiple 3D point clouds, which are captured using one or more 3D cameras, are obtained. At least two of the 3D point clouds correspond to different positions of a body relative to at least a single one of the one or more 3D cameras. Two or more of the 3D point clouds are identified as corresponding to two or more predefined poses, and a segmented representation of the body is generated, in accordance with a 3D part-based volumetric model including cylindrical representations, based on the two 3D point clouds identified as corresponding to the two predefined pose.

First claim

What is claimed is:

1. A method performed by a computer system comprising processor electronics and at least one memory device, the method comprising:
obtaining multiple three dimensional (3D) point clouds captured using one or more 3D cameras, wherein at least two of the 3D point clouds correspond to different positions of a body relative to at least a single one of the one or more 3D cameras;
identifying two of the 3D point clouds as corresponding to two predefined poses, wherein the identifying comprises looking for a static frame using a pixel-wise absolute difference map between two consecutive depth maps for the 3D point clouds; and
generating, based on the two 3D point clouds identified as corresponding to the two predefined poses, a segmented representation of the body in accordance with a 3D part-based volumetric model comprising cylindrical representations.

2. The method of claim 1 , wherein:
the identifying comprises identifying four of the 3D point clouds as corresponding to four predefined poses, including a forward pose, a left facing pose, a backward pose, and a right facing pose, and wherein the identifying comprises using a bounding box in depth maps for the 3D point clouds to identify the left facing pose and the right facing pose; and
the generating comprises generating, based on the four 3D point clouds identified as corresponding to the four predefined poses, the segmented representation of the body in accordance with the 3D part-based volumetric model comprising cylindrical representations.

3. The method of claim 2 , wherein the one or more 3D cameras is a single 3D camera.

4. The method of claim 2 , wherein the obtaining comprises capturing the multiple 3D point clouds using the one or more 3D cameras.

5. The method of claim 2 , wherein the identifying comprises automatically identifying the 3D point clouds without user input corresponding to the predefined poses.

6. The method of claim 2 , wherein the generating comprises:
registering the four 3D point clouds with each other; and
building an initial model of the body using the four registered 3D point clouds.

7. The method of claim 6 , wherein the generating comprises refining the initial model using one or more 3D point clouds other than the four 3D point clouds identified as corresponding to the four predefined poses.

8. The method of claim 6 , wherein the registering comprises:
registering the two 3D point clouds corresponding to the forward pose and the left facing pose;
registering the two 3D point clouds corresponding to the forward pose and the right facing pose; and
registering the two 3D point clouds corresponding to the left facing pose and the backward pose.

9. The method of claim 1 , wherein the generating comprises using iterative local registration of limbs and a torso for the body.

10. The method of claim 1 , wherein the segmented representation comprises a segmented mesh representing the body.

11. A non-transitory computer-readable medium, which encodes a computer program product that is operable to cause data processing apparatus to perform operations comprising:
obtaining multiple three dimensional (3D) point clouds captured using one or more 3D cameras, wherein at least two of the 3D point clouds correspond to different positions of a body relative to at least a single one of the one or more 3D cameras;
identifying two of the 3D point clouds as corresponding to two predefined poses, wherein the identifying comprises looking for a static frame using a pixel-wise absolute difference map between two consecutive depth maps for the 3D point clouds; and
generating, based on the two 3D point clouds identified as corresponding to the two predefined poses, a segmented representation of the body in accordance with a 3D part-based volumetric model comprising cylindrical representations.

12. The non-transitory computer-readable medium, of claim 11 , wherein:
the identifying comprises identifying four of the 3D point clouds as corresponding to four predefined poses, including a forward pose, a left facing pose, a backward pose, and a right facing pose, and wherein the identifying comprises using a bounding box in depth maps for the 3D point clouds to identify the left facing pose and the right facing pose; and
the generating comprises generating, based on the four 3D point clouds identified as corresponding to the four predefined poses, the segmented representation of the body in accordance with the 3D part-based volumetric model comprising cylindrical representations.

13. The non-transitory computer-readable medium, of claim 12 , wherein the one or more 3D cameras is a single 3D camera, the obtaining comprises capturing the multiple 3D point clouds using the single 3D camera, and the identifying comprises automatically identifying the 3D point clouds without user input corresponding to the predefined poses.

14. The non-transitory computer-readable medium, of claim 12 , wherein the generating comprises:
registering the four 3D point clouds with each other; and
building an initial model of the body using the four registered 3D point clouds.

15. The non-transitory computer-readable medium, of claim 14 , wherein the generating comprises refining the initial model using one or more 3D point clouds other than the four 3D point clouds identified as corresponding to the four predefined poses.

16. The non-transitory computer-readable medium, of claim 14 , wherein the registering comprises:
registering the two 3D point clouds corresponding to the forward pose and the left facing pose;
registering the two 3D point clouds corresponding to the forward pose and the right facing pose; and
registering the two 3D point clouds corresponding to the left facing pose and the backward pose.

17. The non-transitory computer-readable medium, of claim 11 , wherein the generating comprises using iterative local registration of limbs and a torso for the body, and the segmented representation comprises a segmented mesh representing the body.

18. A system comprising:
a user interface device; and
one or more computers operable to interact with the user interface device, the one or more computers comprising at least one processor and at least one memory device, and the one or more computers programmed to perform operations comprising: obtaining multiple three dimensional (3D) point clouds captured using one or more 3D cameras, wherein at least two of the 3D point clouds correspond to different positions of a body relative to at least a single one of the one or more 3D cameras; identifying two of the 3D point clouds as corresponding to two predefined poses, wherein the identifying comprises looking for a static frame using a pixel-wise absolute difference map between two consecutive depth maps for the 3D point clouds; and generating, based on the two 3D point clouds identified as corresponding to the two predefined poses, a segmented representation of the body in accordance with a 3D part-based volumetric model comprising cylindrical representations.

19. The system of claim 18 , wherein the one or more computers comprise a server operable to interact with the user interface device through a data communication network, and the user interface device is operable to interact with the server as a client.

20. The system of claim 18 , wherein the identifying comprises identifying four of the 3D point clouds as corresponding to four predefined poses, including a forward pose, a left facing pose, a backward pose, and a right facing pose, wherein the iden

Assignees

• Univ Southern California

Inventors

• Medioni Gerard Guy
• Choi Jongmoo
• Wang Ruizhe

Classifications

• G06T2210/12

  Bounding box · CPC title

• G06T7/00 Primary

  Image analysis · CPC title

• G06T17/10

  Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title