Caching and updating of dense 3D reconstruction data

Patent metadata

Field	Value
Publication number	US-10713852-B2
Application number	US-201816229372-A
Country	US
Kind code	B2
Filing date	Dec 21, 2018
Priority date	Dec 22, 2017
Publication date	Jul 14, 2020
Grant date	Jul 14, 2020

Abstract

A method to efficiently update and manage outputs of real time or offline 3D reconstruction and scanning in a mobile device having limited resource and connection to the Internet is provided. The method makes available to a wide variety of mobile XR applications fresh, accurate and comprehensive 3D reconstruction data, in either single user applications or multi-user applications sharing and updating the same 3D reconstruction data. The method includes a block-based 3D data representation that allows local update and maintains neighbor consistency at the same time, and a multi-layer caching mechanism that retrieves, prefetches, and stores 3D data efficiently for XR applications.

First claim

What is claimed is:

1. A portable electronic system comprising:
a sensor configured to capture three-dimensional (3D) information about objects in a physical world;
a local memory;
a transceiver configured for communication over a computer network with remote memory;
a processor configured to execute computer executable instructions to provide a 3D representation of a portion of the physical world based at least in part on the 3D information about the objects in the physical world, wherein:
the 3D representation of the portion of the physical world comprises a plurality of blocks, the plurality of blocks having versions, each version having values representing objects in a region of portion of the physical world at a point in time; and
the computer executable instructions comprise instructions for:
identifying a subset of the plurality of blocks corresponding to the portion of the physical world; and
selecting versions of blocks representing the subset of blocks by (i) accessing previously stored versions of blocks in the local memory or (ii) accessing previously stored versions of blocks in the remote memory and/or (iii) generating new versions of blocks based at least in part on the 3D information about the objects in the physical world.

2. The portable electronic system of claim 1 , wherein the processor implements a service that provides the 3D representation of the portion of the physical world to an application.

3. The portable electronic system of claim 2 , wherein the application executes on the portable electronic system.

4. The portable electronic system of claim 2 , wherein the computer executable instructions for selecting versions of blocks comprise instructions for determining which of the selected versions of the blocks to provide to the application.

5. The portable electronic system of claim 2 , wherein the service pages out of the memory to a local cache or cloud cache versions of blocks corresponding to part of the physical world for which a metric indicative of change is below a threshold.

6. The portable electronic system of claim 1 , wherein selecting versions of blocks representing the subset of blocks comprises selecting a previously stored version of a block in the local memory when a magnitude of a block metric is below a threshold value since the version was stored.

7. The portable electronic system of claim 6 , wherein the magnitude of the block metric indicates a degree of geometry changes of the block.

8. The portable electronic system of claim 6 , wherein the block metric is a timestamp.

9. The portable electronic system of claim 1 , wherein selecting versions of blocks representing the subset of blocks comprises selecting a previously stored version of a block in the remote memory when there is no corresponding version of the block in the local memory.

10. The portable electronic system of claim 1 , wherein selecting versions of blocks representing the subset of blocks comprises generating new versions of a block based at least in part on 3D information about the objects in the physical world when there is no corresponding up to date version of the block in the local memory or the remote memory.

11. The portable electronic system of claim 1 , wherein generating new versions of blocks comprises generating boundary values of a block based on values of adjoining blocks.

12. The portable electronic system of claim 1 , wherein the processor provides information through an application programming interface (API) executing on the portable electronic system.

13. The portable electronic system of claim 1 , wherein the computer executable instructions further comprise instructions for:
creating a coordinate frame in the portion of the physical world, and
wherein each block of the plurality of blocks represent objects in a different region of the portion of the physical world that is identifiable using the coordinate frame.

14. The portable electronic system of claim 1 , wherein:
the computer executable instructions further comprise instructions for:
processing the 3D information into voxels, bricks, and tiles, each voxel comprising one or more signed distance functions (SDFs), each brick comprising one or more voxels, and each tile comprising one or more bricks, and
generating each block of the plurality of blocks by converting one or more tiles into meshes; and

the 3D information is paged out of the local memory on the basis of tiles and/or blocks.

Assignees

• Magic Leap Inc

Inventors

• Molyneaux David Geoffrey
• Steinbrucker Frank Thomas
• Wu Zhongle
• Wei Xiaolin
• Min Jianyuan
• Zhang Yifu

Classifications

• G06T19/00 Primary

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• H04N7/18

  Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast · CPC title

• G06F3/01

  Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence) · CPC title

• G02B27/01

  Head-up displays · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title