Reconstruction of articulated objects from a moving camera

Patent metadata

Field	Value
Publication number	US-9747668-B2
Application number	US-201615003650-A
Country	US
Kind code	B2
Filing date	Jan 21, 2016
Priority date	Jan 21, 2016
Publication date	Aug 29, 2017
Grant date	Aug 29, 2017

Abstract

Systems and method for the reconstruction of an articulated object are disclosed herein, The articulated object can be reconstructed from image data collected by a moving camera over a period of time. A plurality of 2D feature points can be identified within the image data. These 2D feature points can be converted into three-dimensional space, which converted points are identified as 3D feature points. These 3D feature points can be used to identify one or several rigidity constrains and/or kinematic constraints. These rigidity and/or kinematic constraints can be applied to a model of the reconstructed articulated object.

First claim

What is claimed is:

1. A method of reconstructing an articulated object from images captured with a moving camera, the method comprising:
generating a plurality of 2D tracks, wherein at least one of the 2D tracks of the plurality of 2D tracks comprises a plurality of 2D feature points, wherein one of the 2D feature points identifies a two-dimensional location of a unique one of a series of points on an articulated object in a one of a sequenced series of captured images, wherein the sequenced series of captured images are captured by a moving camera, and wherein the moving camera is moving with respect to at least a fixed portion of a background of the images forming the sequenced series of captured images;
generating a manifold of rays comprising a plurality of rays, wherein the rays of the manifold of rays are defined in part as extending through a unique one of the data points of one of the 2D tracks;
converting at least some of the plurality of 2D data points of the plurality of 2D tracks to 3D data points by identifying a position for at least one of the plurality of 2D data points along one of the plurality of rays forming the manifold of rays; and
generating a first 3D representation of the articulated object for each of the images of the sequenced series of captured images, wherein the first 3D representation comprises the 3D data points converted from 2D data points gathered from one of the images of the sequenced series of captured images.

2. The method of claim 1 , wherein the plurality of 2D data points together identify a 2D location of the unique one of the series of points of the articulated object in each of the sequenced series of captured images.

3. The method of claim 1 , wherein generating the manifold of rays comprises:
identifying a lens center for a lens of the moving camera for each of the plurality of the images forming the sequenced series of captured images; and
generating a plurality of rays, wherein a ray extends through a pair of a lens center for one of the images forming the sequenced series of captured images and at least one 2D feature point for that one of the images forming the sequenced series of captured images.

4. The method of claim 3 , wherein a unique ray is generated for each pair of one lens center and one 2D feature point for each of the images forming the sequenced series of images.

5. The method of claim 3 , wherein generating a manifold of rays comprises generating a plurality of manifolds or rays, wherein one of manifolds of rays forming the plurality of manifolds of rays is generated for each of the plurality of 2D tracks.

6. The method of claim 5 , wherein identifying a position for at least some of the plurality of 2D data points along the each of the plurality of rays forming the manifold of rays comprises:
generating a first energy function the result of which is based on the magnitude of difference in the positions of temporally adjacent 3D data points along their respective ray; and
adjusting the position of the 3D data points to minimize the result of the first energy function.

7. The method of claim 6 , further comprising refining the first 3D representation of the articulated object by applying piecewise rigidity constraints, wherein applying piecewise rigidity constrains comprises:
identifying first and second groups of 3D data points as belonging to common first and second rigid members;
generating first and second rigid objects corresponding to the identified groups of 3D data points belonging to the first and second common rigid members;
determining the rotation and translation of the first and second rigid objects from one image of the sequenced series of captured images to a temporally adjacent image of the sequence of captured images;
generating a second 3D representation of the articulated object for each of the images based on the generated first and second rigid objects and the determined rotation and translation of the generated first and second rigid objects;
determining the error between the second 3D representation and the 2D tracks; and
refining the determined rotation and translation of the first and second rigid objects to minimize the error between the second 3D representation and the 2D tracks.

8. The method of claim 7 , wherein refining the determined rotation and translation of the first and second rigid objects to minimize the error between the second 3D representation and the 2D tracks comprises adjusting the determined error with a smoothness parameter.

9. The method of claim 7 , further comprising generating a third 3D representation of the articulated object by applying kinematic constraints to the second 3D representation.

10. The method of claim 9 , wherein applying kinematic constraints comprises:
determining the distance between the first and second rigid members for each image of the sequenced series of captured images;
determining the variation in the determined distances between the first and second rigid members; and
identifying a joint if the determined distance between the first and second rigid members and the variation in the determined distances between the first and second rigid members indicate the presence of a joint.

11. The method of claim 9 , wherein applying kinematic constraints comprises:
determining the position of the first rigid member based on the rotation and translation of the first rigid member;
determining the position of the second rigid member based on the rotation and translation of the second rigid member and the determined position of the first rigid member;
generating the third 3D representation based on the determined position of the first and second rigid members;
determining the error between the third 3D representation and the 3D tracks;
refining the determined positions of the first and second rigid members to minimize the error between the third 3D representation and the 2D tracks.

12. A system for reconstructing an articulated object from images captured with a moving camera, the method comprising:
memory comprising stored instructions; and
a processor configured to:
generate a plurality of 2D tracks, wherein at least one of the 2D tracks of the plurality of 2D tracks comprises a plurality of 2D data points identifying 2D locations of a unique one of a series of points on an articulated object in a one of a sequenced series of captured images, wherein the sequenced series of captured images are captured by a moving camera, and wherein the moving camera is moving with respect to at least a fixed portion of a background of the images forming the sequenced series of captured images;
generate a manifold of rays comprising a plurality of rays, wherein the rays of the manifold of rays are defined in part as extending through a unique one of the data points of one of the 2D tracks;
convert at least some of the plurality of 2D data points of the plurality of 2D tracks to 3D data points by identifying a position for at least some of the plurality of 2D data points along the each of the plurality of rays forming the manifold of rays; and
generate a first 3D representation of the articulated object for each of the images of the sequenced series of captured images, wherein the first 3D representation comprises the 3D data points converted from 2D data points gathered from one of the images of the sequenced series of captured images.

13. The system of claim 12 , wherein the plurality of 2D data points together identify a 2D location of the unique one of the series of points of the articulated object in each of the sequenced series of captured images.

14. The system of claim 12 , wherein gener

Assignees

• Disney Entpr Inc
• Eth Zürich (Eidgenöessische Technische Hochschule Zürich)

Inventors

• Yücer Kaan
• Sorkine Hornung Alexander
• Wang Oliver
• Sorkine Hornung Olga

Classifications

• G06T7/004

  Physics · mapped topic

• G06T5/00 Primary

  Image enhancement or restoration · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T7/0079

  Physics · mapped topic

• G06T2207/20144

  Physics · mapped topic