Methods and systems for detecting and combining structural features in 3D reconstruction

Patent metadata

Field	Value
Publication number	US-11189093-B2
Application number	US-201916719832-A
Country	US
Kind code	B2
Filing date	Dec 18, 2019
Priority date	Sep 25, 2015
Publication date	Nov 30, 2021
Grant date	Nov 30, 2021

Abstract

A method for forming a reconstructed 3D mesh includes receiving a set of captured depth maps associated with a scene, performing an initial camera pose alignment associated with the set of captured depth maps, and overlaying the set of captured depth maps in a reference frame. The method also includes detecting one or more shapes in the overlaid set of captured depth maps and updating the initial camera pose alignment to provide a shape-aware camera pose alignment. The method further includes performing shape-aware volumetric fusion and forming the reconstructed 3D mesh associated with the scene.

First claim

What is claimed is:

1. A method of detecting a shape present in a scene, the method comprising:
determining, using one or more processors and a plurality of point normals, a vertical direction associated with a point cloud including a plurality of captured depth maps;
forming, using the one or more processors, a virtual plane orthogonal to the vertical direction;
projecting, using the one or more processors, the points of the point cloud onto the virtual plane;
calculating, using the one or more processors, projection statistics for the points of the point cloud;
detecting, using the one or more processors, one or more lines from the calculated projection statistics, the one or more lines being associated with vertical walls; and
detecting, using the one or more processors, the shape present in the scene from the projection statistics and the one or more detected lines.

2. The method of claim 1 further comprising determining dimensions and positions of the detected shape.

3. The method of claim 1 wherein determining the vertical direction comprises, for a plurality of pixels in the point cloud:
determining a plurality of horizontal planes defined, for each of the plurality of pixels, by neighboring pixels to each of the plurality of pixels; and
computing a vector normal to the plurality of horizontal planes.

4. The method of claim 1 wherein the projection statistics comprise a number of points of the point cloud projected onto a predetermined x,y location in the virtual plane.

5. The method of claim 4 wherein the projection statistics comprise a distribution of point normals for the points of the point cloud projected onto the predetermined x,y location in the virtual plane.

6. The method of claim 4 wherein the projection statistics comprise an initial height of the points of the point cloud projected onto the predetermined x,y location in the virtual plane.

7. The method of claim 1 wherein the detected shape present in the scene comprises a wall corner.

8. The method of claim 1 wherein the detected shape present in the scene comprises a door frame.

9. The method of claim 1 wherein the plurality of captured depth maps are each obtained from different positions with respect to a scene.

10. The method of claim 1 wherein the plurality of captured depth maps are each obtained from a single position with respect to a scene at different times.

11. The method of claim 1 further comprising forming the point cloud by overlaying the plurality of captured depth maps in a reference frame.

12. The method of claim 1 wherein the vertical direction is aligned with a gravity vector.

13. The method of claim 1 wherein determining the vertical direction comprises:
estimating the vertical direction; and
refining the estimated vertical direction.

14. The method of claim 1 wherein projecting the points of the point cloud onto the virtual plane comprises forming a two-dimensional data set.

Assignees

• Magic Leap Inc

Inventors

• Wei Xiaolin
• Zhang Yifu

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/20068

  Projection on vertical or horizontal image axis · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T7/50

  Depth or shape recovery · CPC title