Mixed three dimensional scene reconstruction from plural surface models

Patent metadata

Field	Value
Publication number	US-9646410-B2
Application number	US-201514788722-A
Country	US
Kind code	B2
Filing date	Jun 30, 2015
Priority date	Jun 30, 2015
Publication date	May 9, 2017
Grant date	May 9, 2017

Abstract

A three-dimensional (3D) scene is computationally reconstructed using a combination of plural modeling techniques. Point clouds representing an object in the 3D scene are generated by different modeled techniques and each point is encoded with a confidence value which reflects a degree of accuracy in describing the surface of the object in the 3D scene based on strengths and weaknesses of each modeling technique. The point clouds are merged in which a point for each location on the object is selected according to the modeling technique that provides the highest confidence.

First claim

What is claimed is:

1. A method of modeling a three-dimensional object from plural image data sources, the method comprising:
providing a first point cloud including a first plurality of points defined in space, the first plurality of points being derived using a multiview stereo (MVS) process from a first one or more images of the object, the first one or more images being of a first image type, each point in the first plurality representing a location on a surface of the three-dimensional object, and each point in the first plurality having a first confidence value on a first confidence scale, the first confidence value being based upon an average texturedness value of the first one or more images;
providing a second point cloud including a second plurality of points defined in space, the second plurality of points being derived from a second one or more images of the object, the second one or more images being of a second image type, each point in the second plurality representing a location on the surface of the three-dimensional object, and each point in the second plurality having a second confidence value on a second confidence scale;
merging the first plurality and the second plurality of points into a third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including:
normalizing one of the first or second confidence scales with the respective second or first confidence scale; and
for each location of the object for which a corresponding point exists in both the first point cloud and the second point cloud, selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud having a greater first or second normalized confidence value.

2. The method according to claim 1 , further comprising:
merging the first plurality and the second plurality of points into the third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud in response to the first or second normalized confidence value being greater than a predetermined threshold.

3. A computing device configured for modeling a three-dimensional object from plural image data sources, the computing device comprising:
one or more processors;
a network interface for supporting communications with the rendering device; and
one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to perform a method for controlling access to data from the remote client device comprising the steps of:
provide a first point cloud including a first plurality of points defined in space, the first plurality of points being derived using a multiview stereo (MVS) process from a first one or more images of the object, the first one or more images being of a first image type, each point in the first plurality representing a location on a surface of the three-dimensional object, and each point in the first plurality having a first confidence value on a first confidence scale, the first confidence value being based upon an average texturedness value of the first one or more images;
provide a second point cloud including a second plurality of points defined in space, the second plurality of points being derived from a second one or more images of the object, the second one or more images being of a second image type, each point in the second plurality representing a location on the surface of the three-dimensional object, and each point in the second plurality having a second confidence value on a second confidence scale;
merge the first plurality and the second plurality of points into a third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including:
normalizing one of the first and second confidence scales with the respective second or first confidence scale; and
for each location of the object for which a corresponding point exists in both the first point cloud and the second point cloud, selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud having a greater first or second normalized confidence value.

4. The computing device according to claim 3 , further comprising:
the one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to merge the first plurality and the second plurality of points into the third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud in response to the first or second normalized confidence value being greater than a predetermined threshold.

5. A system for modeling a three-dimensional object from plural image data sources, the system comprising:
first and second image capture devices, the first and second image capture devices each being operative to produce image data representing an image of the three-dimensional object, at least one of the first or second image capture devices further being operative to produce image data representing a silhouette image of the object; and
a computing device configured for modeling a three-dimensional object from plural image data sources, the computing device including
one or more processors;
a network interface for supporting communications with the rendering device; and
one or more memories storing computer-readable instructions which, when executed by the one or more processors, cause the one or more processors to perform a method for modeling a three-dimensional object comprising the steps of:
provide a first point cloud including a first plurality of points defined in space, the first plurality of points being derived using a multiview stereo (MVS) process from a first one or more images of the object, the first one or more images being of a first image type, each point in the first plurality representing a location on a surface of the three-dimensional object, and each point in the first plurality having a first confidence value on a first confidence scale, the first confidence value being based upon an average texturedness value of the first one or more images;
provide a second point cloud including a second plurality of points defined in space, the second plurality of points being derived from a second one or more images of the object, the second one or more images being of a second image type, each point in the second plurality representing a location on the surface of the three-dimensional object, and each point in the second plurality having a second confidence value on a second confidence scale;
merge the first plurality and the second plurality of points into a third merged point cloud, each point in the third merged point cloud representing a location on the surface of the object, including:
normalizing one of the first and second confidence scales with the respective second or first confidence scale; and
for each location of the object for which a corresponding point exists in both the first point cloud and the second point cloud, selecting the point for inclusion in the merged point cloud from either the first point cloud or the second point cloud having a greater first or second normalized confidence value.

6. A method of modeling a three-dimensional object from plural image data sources, the method comprising:
providing a first point cloud including a first plurality of p

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Collet Romea Alvaro
• Sullivan Steve
• Kirk Adam

Classifications

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06V20/653 Primary

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title

• G06F18/253

  of extracted features · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title