Viewpoint dependent brick selection for fast volumetric reconstruction

Patent metadata

Field	Value
Publication number	US-12299828-B2
Application number	US-202318150078-A
Country	US
Kind code	B2
Filing date	Jan 4, 2023
Priority date	Dec 22, 2017
Publication date	May 13, 2025
Grant date	May 13, 2025

Abstract

A method to culling parts of a 3D reconstruction volume is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data with low usage of computational resources and storage spaces. The method includes culling parts of the 3D reconstruction volume against a depth image. The depth image has a plurality of pixels, each of which represents a distance to a surface in a scene. In some embodiments, the method includes culling parts of the 3D reconstruction volume against a frustum. The frustum is derived from a field of view of an image sensor, from which image data to create the 3D reconstruction is obtained.

First claim

What is claimed is:

1. A method of operating an electronic system to provide information about objects in a physical world, wherein the electronic system comprises active memory and cache memory, the electronic system represents a 3D reconstruction of the physical world as a plurality of blocks, the electronic system comprises a portable, pointable component, based on actions of a user, at a portion of the physical world, and the portable, pointable component comprises at least one sensor, the method comprising:
computing, based at least in part on an output of the at least one sensor, a region of the physical world at which the portable, pointable component will be pointed at a future time; and
selecting blocks of the plurality of blocks based on the computed region.

2. The method of claim 1 , further comprising:
updating information stored in the active memory such that the active memory stores the selected blocks.

3. The method of claim 2 , wherein updating information stored in the active memory comprises moving from the active memory to the cache memory blocks that were not selected.

4. The method of claim 1 , wherein:
the electronic system implements a service for supplying 3D reconstruction data to applications through an application interface, and
the method further comprises selecting blocks from the active memory to supply an application through an application programming interface (API) based on changes to each of the blocks since a version of the block was last supplied to the application.

5. The method of claim 1 , further comprising:
requesting the selected blocks from a remote memory of a computer network; and
updating information stored in the cache memory such that the cache memory stores the selected blocks.

6. The method of claim 1 , wherein the computing is based at least in part on translation and rotation of the head of a user.

7. The method of claim 1 , wherein the computing is based at least in part on objects in the scene.

8. The method of claim 1 , further comprising updating information stored in a local cache such that the local cache stores the selected blocks.

9. A system providing a cloud-based service for a plurality of portable electronic devices that track their respective locations with respect to a 3D environment, the system comprising:
a communication interface configured to communicate with the plurality of portable electric devices, wherein the cloud service is configured to
receive through the communication interface from one or more of the portable electronic devices dense representations of regions of the 3D environment;
store the received dense representations as a stored representation of the 3D environment;
receive through the communication interface from a portable device of the plurality of portable electronic devices an indication of a location of the device within the 3D environment;
retrieve from the stored representation a dense representation of a portion of the 3D environment selected based on the received indication of the location; and
communicate through the communication interface the retrieved dense representation of the selected portion of the 3D environment.

10. The system of claim 9 , wherein the cloud service is in a cache.

11. The system of claim 9 , wherein the dense representation of the selected portion is a mesh block.

12. The system of claim 9 , wherein the cloud service is configured to receive through the communication interface from the one or more of the portable electronic devices dense representations of regions of the 3D environment when respective field-of-views (FOVs) of the one or more of the portable electronic devices change.

13. A system providing a cloud-based service for a plurality of portable electronic devices that track their respective locations with respect to a 3D environment and represent a portion of the 3D environment within their field-of-views (FOVs) with respect to their respective locations as blocks of 3D reconstruction data, each block of 3D reconstruction data corresponds to a predetermined volume of the 3D environment, the system comprising:
a communication interface configured to communicate with the plurality of portable electric devices, wherein the cloud service is configured to
receive through the communication interface from one or more of the portable electronic devices at least one block of 3D reconstruction data when respective FOVs of the one or more of the portable electronic devices change, and
persist the at least one block of 3D reconstruction data as a persisted representation of the 3D environment.

14. The system of claim 13 , wherein the at least one block of 3D reconstruction data is in the format of meshes, pointe clouds, or voxels.

15. The system of claim 13 , wherein the cloud service is in a cache.

16. The system of claim 13 , wherein the cloud service is configured to
receive through the communication interface from a portable device of the plurality of portable electronic devices an indication of a location of the device within the 3D environment;
retrieve from the persisted representation one or more blocks of 3D reconstruction data selected based on the received indication of the location; and
communicate through the communication interface the retrieved one or more blocks of 3D reconstruction data.

17. The system of claim 13 , wherein the cloud service is configured to merge two or more blocks of 3D reconstruction data from the one or more of the portable electronic devices based on a same coordinate system.

Assignees

• Magic Leap Inc

Inventors

• Steinbrücker Frank Thomas
• Molyneaux David Geoffrey
• Wu Zhongle
• Wei Xiaolin
• Min Jianyuan
• Zhang Yifu

Classifications

• G06T2210/12

  Bounding box · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T17/205

  Re-meshing · CPC title

• G06T7/0002

  Inspection of images, e.g. flaw detection · CPC title

• G02B27/017 Primary

  Head mounted · CPC title