Viewpoint dependent brick selection for fast volumetric reconstruction

Patent metadata

Field	Value
Publication number	US-11580705-B2
Application number	US-202117240301-A
Country	US
Kind code	B2
Filing date	Apr 26, 2021
Priority date	Dec 22, 2017
Publication date	Feb 14, 2023
Grant date	Feb 14, 2023

Abstract

A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.

First claim

What is claimed is:

1. A computing device configured to generate a three-dimensional (3D) reconstruction of a portion of a scene, the computing device comprising:
a support member;
a depth sensor coupled to the support member;
a central processing unit (CPU) coupled to receive a depth map from the depth sensor; and
at least one non-transitory computer-readable medium comprising computer-executable instructions configured for execution on the CPU that, when executed by the CPU, perform a method comprising:
generating or updating, based at least in part on the depth map, a grid of voxels corresponding to the portion of the scene, each voxel comprising a signed distance field (SDF) value that indicates a distance from a corresponding region in the portion of the scene to the region's closes surface in the portion of the scene;
extracting plane data based at least in part on a plurality of bricks, each brick comprising a portion of the grid of voxels and being identifiable by a brick ID; and
storing the plane data in a plane data store.

2. The computing device of claim 1 , wherein the brick ID for a brick indicates a location of the brick in the portion of the scene.

3. The computing device of claim 1 , wherein the plane data comprises surfels comprising vertices of meshes for the plurality of bricks.

4. The computing device of claim 1 , wherein extracting the plane data comprises extracting at most one brick plane for each of the plurality of bricks.

5. The computing device of claim 1 , wherein extracting the plane data comprises:
extracting brick planes for the plurality of bricks; and
generating global planes, each global plane comprising a plurality of brick planes.

6. The computing device of claim 5 , wherein extracting the plane data further comprises assigning plane IDs to the brick planes and global planes.

7. The computing device of claim 5 , wherein extracting brick planes for the plurality of bricks comprises:
generating brick meshes for the plurality of bricks;
dividing at least one of the brick meshes into a plurality of sub-brick meshes;
detecting sub-brick planes based on the plurality of sub-brick meshes; and
extracting brick planes for the bricks corresponding to the at least one of the brick meshes based on the detected sub-brick planes.

8. The computing device of claim 7 , wherein the method further comprises updating the brick meshes based on the plane data such that the updated brick meshes are less noisy.

9. The computing device of claim 1 , wherein:
the plane data store comprises, prior to receiving the depth map, existing brick planes for at least a portion of the plurality of bricks and existing global planes each comprising a plurality of the existing brick planes, and
storing the plane data in the plane data store comprises:
updating existing brick planes and existing global planes in the plane data store with the plane data.

10. The computing device of claim 9 , wherein updating existing brick planes and existing global planes in the plane data store comprises adding and/or removing and/or replacing a portion of the existing brick planes based on the plane data.

11. The computing device of claim 9 , wherein updating existing brick planes and existing global planes in the plane data store comprises merging and/or splitting a portion of the existing global planes based on the plane data.

12. A method of operating a computing system to generate a three-dimensional (3D) representation of a portion of a scene, the method comprising:
receiving a query from an application requesting a planar geometry representation;
searching a plane data store for plane data corresponding to the query;
generating a rasterized plane mask from the plane data corresponding to the query, the rasterized plane mask comprising a plurality of plane coverage points;
generating the 3D representation of the portion of the scene based at least in part on the rasterized plane mask according to the requested planar geometry representation; and
sending the generated 3D representation of the portion of the scene to the application.

13. The method of claim 12 , wherein the query indicates at least one of a location, a size, or an orientation of a plane.

14. The method of claim 12 , wherein the requested planar geometry representation is selected from a group comprising outer rectangular planes, inner rectangular planes, and polygon planes.

15. The method of claim 12 , wherein generating the rasterized plane mask comprises generating the plane coverage points by projecting boundary points of brick planes onto global planes.

16. The method of claim 12 , wherein generating the 3D representation of the portion of the scene based at least in part on the rasterized plane mask according to the requested planar geometry representation comprises:
when the requested planar geometry representation is outer rectangular planes,
generating an outer rectangle that is the smallest rectangle surrounding the rasterized plane mask.

17. The method of claim 12 , wherein generating the 3D representation of the portion of the scene based at least in part on the rasterized plane mask according to the requested planar geometry representation comprises:
when the requested planar geometry representation is inner rectangular planes:
generating a rasterized grid by assigning “1”s to bricks with two plane coverage points and “0”s to bricks without two plane coverage points;
determining groups of bricks, each group comprising a plurality of bricks that are marked as “1” and aligned in a line parallel to an edge of a brick; and
generating inner rectangles for the groups of bricks, each being the smallest rectangle that surrounds a respective group.

18. The method of claim 12 , wherein generating the 3D representation of the portion of the scene based at least in part on the rasterized plane mask according to the requested planar geometry representation comprises:
when the requested planar geometry representation is polygon planes,
generating a polygon by connecting at least a portion of the plurality of plane coverage points of the rasterized plane mask such that none of the plurality of plane coverage points is outside the polygon.

Assignees

• Magic Leap Inc

Inventors

• Steinbrücker Frank Thomas
• Molyneaux David Geoffrey
• Wu Zhongle
• Wei Xiaolin
• Min Jianyuan
• Zhang Yifu

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T1/60

  Memory management · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title