Cross reality system for large scale environment reconstruction

Patent metadata

Field	Value
Publication number	US-12223591-B2
Application number	US-202418599083-A
Country	US
Kind code	B2
Filing date	Mar 7, 2024
Priority date	Feb 27, 2020
Publication date	Feb 11, 2025
Grant date	Feb 11, 2025

Abstract

Various techniques pertaining to methods, systems, and computer program products a spatial persistence process that places a virtual object relative to a physical object for an extended-reality display device based at least in part upon a persistent coordinate frame (PCF). A determination is made to decide whether a drift is detected for the virtual object relative to the physical object, upon or after detection of the drift or deviation, the drift or deviation is corrected at least by updating a tracking map into an updated tracking map and further at least by updating the persistent coordinate frame (PCF) based at least in part upon the updated tracking map, wherein the persistent coordinate frame (PCF) comprises six degrees of freedom relative to the map coordinate system.

First claim

What is claimed is:

1. An extended-reality or cross-reality system comprising a wearable system for rendering virtual content, the wearable system further comprising:
a head unit comprising a display system and a plurality of sensor units and located in a physical environment in a first instance of the wearable system in a first session at a first time point; and
a non-transitory computer readable storage medium storing thereupon a plurality of modules, the plurality of modules comprising:
a coordinate system module that comprises a persistent coordinate frame (PCF) system;
a plurality of transform modules that performs one or more transformations pertaining to a local coordinate frame and a camera coordinate frame; and
a processor executing a sequence of instructions, wherein execution of the sequence of instructions causes the processor to perform a set of acts, the set of acts comprising:
persisting, by the persistent coordinate frame system, a virtual content in the physical environment from a first perspective when viewing the virtual content through the wearable system in the first instance of the wearable system in the first session, wherein
the virtual content is spatially persisted in the physical environment to prevent the virtual content from appearing out of place from a second perspective when viewing the virtual content from or through the wearable system in a second instance of the wearable system in a second session at a second time point based at least in part upon a result of the one or more transformations.

2. The extended-reality or cross-reality system of claim 1 , the set of acts further comprising determining the persistent coordinate frame system at least by:
identifying one or more points from the physical environment, wherein the one or more points representing at least one feature recognized by the head unit of the wearable system;
determining at least one degree of freedom of six degrees of freedom of the persistent coordinate frame system based at least in part upon a location at which the persistent coordinate frame system is stored a map coordinate system of the coordinate system module or a canonical map; and
determining the persistent coordinate frame system based at least in part upon the at least one degree of freedom.

3. The extended-reality or cross-reality system of claim 2 , the set of acts further comprising:
identifying the canonical map that is used in determining the at least one degree of freedom of the persistent coordinate frame system, wherein the canonical map includes a canonical map point that corresponds to a feature of an object;
localizing the canonical map to the wearable system for the first session; and
creating a new map at least by stitching the canonical map to the persistent coordinate frame system.

4. The extended-reality or cross-reality system of claim 2 , the set of acts further comprising:
identifying an object; and
representing the object as at least one persistent coordinate frame, wherein
the at least one persistent coordinate frame comprises a local coordinate frame relative to at least one coordinate frame in the plurality of coordinate system module, and
when the object is represented as multiple persistent coordinate frames, the multiple persistent coordinate frames respectively correspond to a plurality of features of the object that is represented as the multiple persistent coordinate frames.

5. The extended-reality or cross-reality system of claim 2 , the set of acts further comprising:
determining a plurality of canonical maps, wherein each canonical map of the plurality of canonical maps respectively represents a corresponding object or a feature of the corresponding object; and
providing a floorplan of a plurality of objects in the physical environment at least by geographically disposing the plurality of canonical maps in a two-dimensional pattern.

6. The extended-reality or cross-reality system of claim 2 , the set of acts further comprising:
determining a plurality of canonical maps;
determining, by one or more sensor units in the head unit of the wearable system, a tracking map that include an image frame, a keyframe, or data derived from the image frame or the keyframe at least by removing redundant information and information with an accuracy or resolution below a threshold from the image frame, the keyframe, or the data derived from the image frame or the keyframe;
ranking the plurality of canonical maps based at least in part upon respective similarities between one or more first regions in the plurality of canonical maps and one or more second regions in the tracking map; and
merging or stitching the tracking map into at least a tile in at least one canonical map of the plurality of canonical maps so as to have a correspondence between a first portion of the tracking map and a second portion of the at least one canonical map.

7. The extended-reality or cross-reality system of claim 6 , merging or stitching the tracking map into at least a tile in the at least one canonical map in the set of acts comprising:
aligning a set of tracking map points with a set of canonical map features in the at least the tile in the at least one canonical map at least by applying a transformation between the set of canonical map features and set of tracking map points; and
determining one or more overlapping portions of the tracking map and at least the tile of the at least one canonical map.

8. The extended-reality or cross-reality system of claim 1 , the set of acts further comprising:
determining, by one or more sensor units in the head unit of the wearable system, a tracking map that include an image frame, a keyframe, or data derived from the image frame or the keyframe at least by removing redundant information and information with an accuracy or resolution below a threshold from the image frame, the keyframe, or the data derived from the image frame or the keyframe;
refining the tracking map into a refined tracking map at least by correcting a drift or a deviation of a tracking path that deviates from an actual tracking path across at least a plurality of sessions that includes the first session;
merging or stitching the tracking map with an environment map, wherein the environment, wherein the environment map stores location or orientation data of at least a portion of the physical environment and data pertaining to gaze directions when perceived with the wearable system, and the environment map comprises more details about the physical world than the tracking map; and
creating or updating, by collaboration among a plurality of wearable systems in multiple sessions in the physical environment, a passable world by using at least the environment map, wherein the passable world so created or updated is shared among the plurality of wearable systems in the multiple sessions including the first and the second sessions.

9. The extended-reality or cross-reality system of claim 1 , the set of acts further comprising attaching the virtual content to the persistent coordinate frame system, attaching the virtual content to the persistent coordinate frame system comprising:
identifying a keyframe from a plurality of keyframes based at least in part upon a set of features in the plurality of image frames captured by one or sensor units in the head unit of the wearable system;
associating a pose of a sensor unit of the one or more sensor units with the keyframe;
generating a persistent pose from the keyframe based at least in part upon the pose and metadata about the keyframe, wherein the persistent pose includes a coordinate location or direction that has one or more associated keyframes; and
reflecting the persistent pose as a persis

Assignees

• Magic Leap Inc

Inventors

• Cao Yilun
• Kandra Mohan Babu
• Molyneaux David Geoffrey
• Olshansky Daniel
• Paul Pena David
• Steinbrücker Frank Thomas
• Torres Rafael Domingos

Classifications

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T7/55

  from multiple images · CPC title