Cross reality system for large scale environment reconstruction

Patent metadata

Field	Value
Publication number	US-11501489-B2
Application number	US-202117185558-A
Country	US
Kind code	B2
Filing date	Feb 25, 2021
Priority date	Feb 27, 2020
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

An extended or cross reality system includes a computing device communicably connected to a plurality of portable electronic devices via a network component, a repository accessible by the computing device and the plurality of portable electronic devices, and a dense map merge component. The extended or cross reality system determines a representation for multiple portions of a 3D environment based at least in part upon on a set of dense maps received from the plurality of portable devices, wherein the set of dense maps is grouped into multiple subgroups based at least in part upon pose data pertaining to the set of dense maps or surface information in the set of dense maps. The extended or cross reality system storing the representation as at least a portion of a shared persistent dense map.

First claim

What is claimed is:

1. An extended reality or cross reality (XR) system for rendering virtual content, comprising:
a computing device communicably connected to a plurality of portable electronic devices via a network component;
a repository accessible by the computing device and the plurality of portable electronic devices; and
a dense map merge component, the extended reality or cross reality system configured to perform a set of acts that comprises:
determining a representation for multiple portions of a 3D environment based at least in part upon on a set of dense maps received from the plurality of portable devices, wherein the set of dense maps is grouped into multiple subgroups based at least in part upon pose data pertaining to the set of dense maps or surface information in the set of dense maps, and
storing the representation as at least a portion of a shared persistent dense map.

2. The extended reality or cross reality (XR) system of claim 1 , the set of acts further comprising:
localizing a sparse map and a dense map to a canonical map that is selected from multiple canonical maps stored in a repository, wherein a coordinate system is derived from the sparse map and is used to determine a position or an orientation of an object in the dense map.

3. The extended reality or cross reality (XR) system of claim 2 , the set of acts further comprising:
determining, for a canonical map, the sparse map comprising headpose data of a headpose of a portable electronic device and object information for one or more objects detected by the portable electronic device at the headposes, wherein the sparse map is oriented local to a local reference frame for the portable electronic device.

4. The extended reality or cross reality (XR) system of claim 2 , the set of acts further comprising:
determining the dense map in a first processing pipeline, wherein the sparse map is determined in a second processing pipeline and is refreshed at a second refresh rate, the dense map is refreshed at a first rate slower than the second refresh rate, and the dense map comprises dense information that references to a persistent pose of a sparse map.

5. The extended reality or cross reality (XR) system of claim 2 , the set of acts further comprising:
promoting the sparse map to a canonical map based at least in part upon a condition that determines an extent of overlap between the sparse map and one or more canonical maps that have been stored in the repository.

6. The extended reality or cross reality (XR) system of claim 1 , wherein the set of acts further comprising:
generating, at the portable electronic device, a shared world model at least by performing a local 3D reconstruction.

7. The extended reality or cross reality (XR) system of claim 6 , generating the shared world model at least by performing the local 3D reconstruction comprising:
identifying, at the portable electronic device of the plurality of portable electronic devices, a pose of a sparse map; and
identifying a shared persistent dense map from a repository accessible by the computing device.

8. The extended reality or cross reality (XR) system of claim 7 , generating the shared world model at least by performing the local 3D reconstruction comprising:
localizing the sparse map generated by the portable electronic device to a canonical map at least by using a transform, the transform determined for the sparse map and the canonical map based at least in part upon one or more persistent coordinate frames in the canonical map.

9. The extended reality or cross reality (XR) system of claim 8 , generating the shared world model at least by performing the local 3D reconstruction comprising:
generating, at the portable electronic device, a dense map at least by determining a local pose data or local depth data of the object or a portion thereof from an image captured by the portable electronic device.

10. The extended reality or cross reality (XR) system of claim 8 , wherein the dense map is generated at the portable electronic device at least further by identifying localized pose data or localized depth data relative to the canonical map, the localized pose data or localized depth data respectively transformed, by the transform, from the localized pose data or local depth data, and transmitting the localized pose data or the localized depth data.

11. The extended reality or cross reality (XR) system of claim 1 , wherein the set of acts further comprising:
performing, at the dense map merge component residing on a remote server, 3D reconstruction of the 3D environment.

12. The extended reality or cross reality (XR) system of claim 1 , performing the 3D reconstruction of the 3D environment comprising:
receiving, from a portable electronic device of the plurality of portable electronic devices, local pose data of a sparse map generated by the portable electronic device; and
identifying a shared persistent dense map from a repository accessible by the dense map merge component.

13. The extended reality or cross reality (XR) system of claim 12 , performing the 3D reconstruction of the 3D environment further comprising:
determining a transform that localizes the sparse map generated by the portable electronic device to a canonical map, wherein the transform is determined for the sparse map and the canonical map based at least in part upon one or more persistent coordinate frames in the canonical map and pose data in the sparse map.

14. The extended reality or cross reality (XR) system of claim 13 , performing the 3D reconstruction of the 3D environment further comprising:
receiving, from the portable electronic device, a dense map, wherein the dense map is generated by using surface at least a portion of the collections of posed surface information and a local pose data or local depth data pertaining to an object or a portion thereof in an image captured by the portable electronic device, the local pose data and the local depth data are respectively transformed into localized pose data or localized depth data relative to the canonical map;
identifying the dense map that corresponds to the sparse map based at least in part upon a localization result of localizing the sparse map to the canonical map; and
computing the representation of the 3D environment at least by merging the dense map to the shared persistent dense map as a part of the representation.

15. The extended reality or cross reality (XR) system of claim 14 , the set of acts further comprising:
localizing a sparse map generated by the portable electronic device of the plurality of portable electronic device to the canonical map; and
generating, at a sparse map merge component, a shared persistent sparse map for the 3D environment.

16. The extended reality or cross reality (XR) system of claim 15 , generating the shared persistent sparse map comprising:
receiving, from a portable electronic device of the plurality of portable electronic devices, local pose data of the object or the portion thereof in a sparse map generated by the portable electronic device, wherein the local pose data is oriented in relation to the portable electronic device, wherein the object or the portion thereof is represented as a point node in the sparse map; and
identifying at least one persistent coordinate frame for the sparse map from the canonical map based at least in part upon the local pose data of the object or the portion thereof.

17. The extended reality or cross reality (XR) system of claim 16 , generating the shared persistent sparse map comprising:
localizing the

Assignees

• Magic Leap Inc

Inventors

• Cao Yilun
• Kandra Mohan Babu
• Molyneaux David Geoffrey
• Olshansky Daniel
• Paul Pena David
• Steinbrücker Frank Thomas
• Torres Rafael Domingos

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/55

  from multiple images · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title