Cross reality system for large scale environment reconstruction

Patent metadata

Field	Value
Publication number	US-11694394-B2
Application number	US-202217949599-A
Country	US
Kind code	B2
Filing date	Sep 21, 2022
Priority date	Feb 27, 2020
Publication date	Jul 4, 2023
Grant date	Jul 4, 2023

Abstract

Various techniques pertaining to methods, systems, and computer program products identify, for a first portable device at a first location in a three-dimensional (3D) physical environment, a first sparse map, the first portable device connected to a remote computing system for accessing sharable world data of the 3D physical environment that comprises a physical object, wherein the plurality electronic devices include the plurality of portable devices having respective sparse maps installed thereupon. A second sparse map is identified for a second portable device connected to the remote computing system for accessing the sharable world data. The second portable device performs content persistence for a virtual content relative to the physical object at least by using a merged map determined by merging a version of the first sparse map and a version of the second sparse map based at least in part upon one or more persistence coordinate frames.

First claim

What is claimed is:

1. A method, comprising:
Identifying, for a first portable device at a first location in a three-dimensional (3D) physical environment, a first sparse map, the first portable device connected to a remote computing system for accessing sharable world data of the 3D physical environment that comprises a physical object, wherein
the plurality electronic devices include the plurality of portable devices having respective sparse maps installed thereupon;

Identifying, for a second portable device at a second location in the 3D physical environment, a second sparse map, the second portable device connected to the remote computing system for accessing the sharable world data; and
performing, by the second portable device, content persistence for a virtual content relative to the physical object at least by using a merged map, wherein the merged map is determined at least by merging a version of the first sparse map and a version of the second sparse map based at least in part upon one or more persistence coordinate frames (PCFs).

2. The method of claim 1 , wherein the remote computing system determines the sharable world data using at least feature data collected by the plurality electronic devices for the sharable world data, and the sharable world data comprises 3D representations for one or more portions of the 3D physical environment.

3. The method of claim 1 , further comprising:
the first portable device entering the first location at a first time point and utilizing the first sparse map as a floorplan for the 3D physical environment, wherein
the first sparse map includes node data pertaining to pose of a node in the first sparse map, and
the node represents a feature of a physical object.

4. The method of claim 3 , further comprising:
deriving a first feature for the first location based at least in part upon first feature data captured by the first portable device.

5. The method of claim 4 , further comprising:
transforming the first feature into a first PCF that comprises a first local coordinate frame for the first portable device for the content persistence; and
the first portable device placing the first PCF at a first pose in the first sparse map for the first location.

6. The method of claim 4 , further comprising:
the first portable device updating the first sparse map into a first updated sparse map using at least the first feature for the first location in the 3D physical environment; and
the first portable device uploading the first updated spars55e map to the remote computing system for the sharable world data.

7. The method of claim 5 , further comprising:
the second portable device entering the second location at a second time point and utilizing the second sparse map as the floorplan;
determining, for the second portable device, that the sharable world data pertaining to the second location has been accessed by one or more other portable devices of the plurality of portable devices; and
the second portable device retrieving one or more nearby PCFs from a plurality of PCFs for the second location from the cloud, wherein the one or more nearby PCFs comprise the first PCF determined by the first portable device.

8. The method of claim 5 , further comprising:
determining a transformation for the second portable device based at least in part upon the first location, the second location, and the first PCF; and
deriving a second feature for the second location based at least in part upon second feature data captured by the second portable device.

9. The method of claim 8 , further comprising:
transforming the second feature into a second PCF that comprises a second local coordinate frame for the second portable device, the second local coordinate frame facilitating for the content persistence;
the second portable device placing the second PCF at a second pose in the second sparse map for the second location of the 3D physical environment; and
updating the second sparse map into a second updated sparse map using at least the first feature for the first location in the 3D physical environment.

10. The method of claim 9 , further comprising:
the second portable device causing an additional PCF to be generated for a third feature in the 3D physical environment; and
expanding the second updated sparse map at least by placing the additional PCF at an additional pose in the second updated sparse map for the sharable world data.

11. The method of claim 7 , further comprising:
the second portable device requesting for one or more sparse maps for the second location; and
merging the second sparse map with the one or more sparse maps.

12. The method of claim 11 , merging the second sparse map with the one or more sparse maps comprising:
merging the second sparse map with a first updated sparse map into a merged sparse map, wherein the first updated sparse map is modified from the first sparse map.

13. The method of claim 12 , merging the second sparse map with the first updated sparse map comprising:
determining whether the first sparse map and the second sparse map overlap; and
when it is determined that the first sparse map and the second sparse map do not overlap,
promoting the first updated sparse map into a first canonical map at least by using the first PCF or the pose data; and
merging the second sparse map with the first updated sparse map at least by merging the second sparse map with the first canonical map.

14. The method of claim 12 , merging the second sparse map with the first updated sparse map comprising:
determining whether the first sparse map and the second sparse map overlap; and
when it is determined that the first sparse map and the second sparse map overlap,
identifying one or more overlapping portions between the first sparse map and the second sparse map; and
merging the second sparse map with the first updated sparse map based at least in part upon the one or more overlapping portion.

15. An extended reality or cross reality (XR) system for rendering virtual content, comprising:
the first portable device that enters a first location in a three-dimensional (3D) physical environment and comprises a first sparse map and a first network component that connects the portable device to a remote computing system for accessing sharable world data of the 3D physical environment that includes a physical object, wherein
the remote computing system determines the sharable world data using at least feature data collected by a plurality of electronic devices for the sharable world data,
the plurality electronic devices include a plurality of portable devices having respective sparse maps installed thereupon,
the plurality of portable devices comprise a second portable device that enters a second location in the 3D physical environment and comprising a second sparse map and a second network component that connects the second portable device to the remote computing system for accessing the sharable world data,
the first portable device performing content persistence for a virtual content relative to the physical object at least by using a merged map, and
the merged map is determined at least by merging a version of the second sparse map and a version of the first sparse map based at least in part upon one or more persistence coordinate frames (PCFs).

16. The XR system of claim 15 , wherein
the first portable device enters the first location at a first time point and utilizes the first sparse map as a floorplan for the 3D physical environment,
the first sparse map includes node data pertaining to pose of

Assignees

• Magic Leap Inc

Inventors

• Cao Yilun
• Kandra Mohan Babu
• Molyneaux David Geoffrey
• Olshansky Daniel
• Paul Pena David
• Steinbrücker Frank Thomas
• Torres Rafael Domingos

Classifications

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T7/55

  from multiple images · CPC title