3d skeletonization using truncated epipolar lines

Patent metadata

Field	Value
Publication number	US-2019139297-A1
Application number	US-201715806145-A
Country	US
Kind code	A1
Filing date	Nov 7, 2017
Priority date	Nov 7, 2017
Publication date	May 9, 2019
Grant date	—

Abstract

Technologies are provided for generating three-dimensional ( 3 D) skeletons of target objects using images of the target objects captured from different viewpoints. Images of an object (such as a person) can be captured from different camera angles. Feature keypoints of the object can be identified in the captured images. Keypoints that identify a same feature in separate images can be correlated using truncated epipolar lines. For example, depth information for a keypoint can be used to truncate an epipolar line that is created using the keypoint. The correlated feature keypoints can be used to create 3 D feature coordinates for the associated features of the object. A 3 D skeleton can be generated using the 3 D feature coordinates. One or more 3 D models can be mapped to the 3 D skeleton and rendered. The rendered one or more 3 D models can be displayed on one or more display devices.

First claim

What is claimed is:

1 . A computing device, comprising:
a processing unit; and a memory; the computing device programmed, via computer-executable instructions, to perform operations for performing three-dimensional skeletonization, the operations comprising:
receiving multiple two-dimensional images of a target object, wherein the multiple two-dimensional images depict the target object from different perspectives;
identifying corresponding keypoints of the target object in the multiple two-dimensional images using one or more truncated epipolar lines;
mapping the corresponding keypoints to three-dimensional coordinates in a three-dimensional space; and generating a three-dimensional skeleton of the target object using the three-dimensional coordinates.

2 . The computing device of claim 1 , wherein:
the target object is a human; and identifying the corresponding keypoints in the multiple two-dimensional images comprises identifying a body part of the human being in two of the two-dimensional images that depict the body part from different perspectives.

3 . The computing device of claim 1 , wherein identifying the corresponding keypoints comprises using the one or more truncated epipolar lines comprises:
projecting an epipolar line on a first of the two-dimensional images using a keypoint in a second of the two-dimensional images; truncating the epipolar line using a depth map associated with the first of the two-dimensional images; and identifying a corresponding keypoint in the first of the two-dimensional images using the truncated epipolar line.

4 . The computing device of claim 1 , wherein the operations further comprise:
mapping a three-dimensional model to at least part of the three-dimensional skeleton; and rendering the three-dimensional model.

5 . The computing device of claim 1 , wherein the mapping the corresponding keypoints to three-dimensional coordinates comprises:
triangulating a three-dimensional coordinate in the three-dimensional space using two corresponding keypoints.

6 . The computing device of claim 5 , wherein the triangulating comprises determining a direct linear transformation using the two corresponding keypoints.

7 . The computing device of claim 1 , wherein identifying the corresponding keypoints comprises:
determining confidence scores for detected keypoints in the multiple two-dimensional images with respect to a feature of the target object; selecting a first keypoint in one of the multiple two-dimensional images with a highest confidence score; selecting a second keypoint in another of the multiple two-dimensional images with a second-highest confidence score; and projecting a truncated epipolar line on the another image using the first keypoint; and determining that the truncated epipolar line intersects the second keypoint.

8 . The computing device of claim 1 , wherein:
the multiple two-dimensional images are captured by multiple cameras configured to view the target object from different perspectives.

9 . The computing device of claim 1 , wherein the operations further comprise:
mapping a three-dimensional point cloud to the three-dimensional skeleton.

10 . A method, implemented by a computing device comprising a processing unit and a memory, the method comprising:
receiving images depicting a person from multiple viewpoints; identifying features of the person in the images; correlating the identified features using one or more truncated epipolar lines; constructing three-dimensional feature coordinates for the identified features using the correlated identified features; and generating a three-dimensional skeleton of the person using the three-dimensional feature coordinates.

11 . The method of claim 10 , further comprising:
generating a point cloud based on depth information captured by one or more depth cameras; mapping the point cloud to the three-dimensional skeleton of the person; and moving one or more points of the point cloud closer to a surface of a volumetric shape associated with one of the three-dimensional feature coordinates.

12 . The method of claim 10 , wherein:
the received images depict multiple people; and the method further comprises:
using the one or more truncated epipolar lines to distinguish between separate features of the multiple people; and
generating separate three-dimensional skeletons for the multiple people.

13 . The method of claim 10 , further comprising:
receiving depth maps associated with the images depicting the person from multiple viewpoints; and generating the one or more truncated epipolar lines using one or more of the received depth maps.

14 . The method of claim 10 , further comprising:
mapping a three-dimensional mesh to the generated three-dimensional skeleton; applying a texture model to at least part of the three-dimensional mesh; and rendering a projection of a three-dimensional scene comprising the three-dimensional mesh with the texture model applied.

15 . The method of claim 10 , wherein the reconstructing three-dimensional feature coordinates for the identified features comprises triangulating positions of the three-dimensional feature coordinates using two-dimensional positions of the correlated features in the received images.

16 . The method of claim 10 , further comprising:
identifying a specified priority region of the person in the received images; and applying more computational resources to identifying and correlating features in the specified priority region than are applied to identifying and correlating features outside the specified priority region.

17 . A system for generating a three-dimensional skeleton of a human, the system comprising:
multiple camera devices configured to capture images of a human from different viewpoints; and a computing device comprising a processor and a storage medium storing executable instructions that, when executed by the processor, program the computing device to perform operations, the operations comprising:
receiving the captured images from the multiple camera devices;
detecting keypoints in the captured images associated with body parts of the human;
correlating the detected keypoints, wherein the correlating comprises using one or more truncated epipolar lines to determine that keypoints in separate images, of the captured images, are associated with a same body part depicted from different viewpoints;
triangulating three-dimensional coordinates for the body parts using the correlated keypoints associated with the body parts; and
generating a three-dimensional skeleton comprising the three-dimensional coordinates.

18 . The system of claim 17 , wherein:
the camera devices comprise depth sensors configured to capture depth maps associated with the captured images; and the operations further comprise:
receiving the depth maps from the camera devices; and
generating the one or more truncated epipolar lines using one or more of the depth maps.

19 . The system of claim 18 , wherein the generating the one or more truncated epipolar lines comprises:
identifying two keypoints in two images, of the captured images, associated with a same body party type; projecting an epipolar line on a first of the two images using the keypoint in a second of the two images; truncating the epipolar line using a depth map associated with the first of the two images; and determining that the truncated epipolar line intersects the keypoint in the first of the two images.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Chen Qiuyu

Classifications

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06V40/10

  Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title