Cross reality system with map processing using multi-resolution frame descriptors

Patent metadata

Field	Value
Publication number	US-12499618-B2
Application number	US-202418609101-A
Country	US
Kind code	B2
Filing date	Mar 19, 2024
Priority date	Feb 13, 2020
Publication date	Dec 16, 2025
Grant date	Dec 16, 2025

Abstract

A distributed, cross reality system efficiently and accurately compares location information that includes image frames. Each of the frames may be represented as a numeric descriptor that enables identification of frames with similar content. The resolution of the descriptors may vary for different computing devices in the distributed system based on degree of ambiguity in image comparisons and/or computing resources for the device. A descriptor computed for a cloud-based component operating on maps of large areas that can result in ambiguous identification of multiple image frames may use high resolution descriptors. High resolution descriptors reduce computationally intensive disambiguation processing. A portable device, which is more likely to operate on smaller maps and less likely to have the computational resources to compute a high resolution descriptor, may use a lower resolution descriptor.

First claim

What is claimed is:

1 . A system for providing shared location-based content, the system comprising:
a portable device, comprising:
at least one processor; and
at least one computer-readable medium comprising instructions that, when executed, cause the at least one processor to:
render virtual content within a representation of a 3D environment;
compute one or more feature descriptors each representing a respective feature of an image of the 3D environment;
send the one or more feature descriptors to a networked resource;
calculate a first frame descriptor from the one or more feature descriptors, where the first frame descriptor has a first resolution less than a second resolution of a second frame descriptor calculated for the image at the networked resource; and
localize the portable device by using the first frame descriptor, wherein the localizing is with respect to at least one of (i) a tracking map generated by the portable device, or (ii) at least a portion of a stored map identified by the network resource based on the second frame descriptor.

2 . The system of claim 1 , further comprising the networked resource, wherein the networked resource comprises at least one processor and at least one computer-readable medium comprising:
a plurality of stored maps of the 3 D environment, wherein at least one stored map of the plurality of stored maps is associated with at least one frame descriptor; and computer executable instructions that, when executed by the at least one processor, cause the at least one processor to:
receive the one or more feature descriptors from the portable device;
calculate the second frame descriptor, the second frame descriptor being representative of the received one or more feature descriptors;
compare the second frame descriptor with the at least one frame descriptor associated with the at least one stored map of the 3 D environment; and
based on comparing the second frame descriptor with the at least one frame descriptor associated with the at least one stored map, localize the portable device to a shared coordinate system.

3 . The system of claim 2 , wherein the computer executable instructions of the networked resource, when executed by the at least one processor of the networked resource, cause the at least one processor of the networked resource to calculate the second frame descriptor using a neural network.

4 . The system of claim 2 , wherein the computer executable instructions of the networked resource, when executed by the at least one processor of the networked resource, further cause the at least one processor of the networked resource to:
receive the tracking map from the portable device; merge the tracking map with the at least one stored map, to produce a merged map comprising location information based on location information of the at least one stored map and the tracking map; and store the merged map in the at least one computer-readable medium of the networked resource.

5 . The system of claim 2 , wherein the second frame descriptor has a resolution greater than 512 bits.

6 . The system of claim 1 , wherein the instructions of the at least one computer-readable medium of the portable device, when executed by the at least one processor of the portable device, cause the at least one processor of the portable device to localize the portable device based on comparing the first frame descriptor to frame descriptors of image frames local to the portable device.

7 . The system of claim 1 , wherein the instructions of the at least one computer-readable medium of the portable device, when executed by the at least one processor of the portable device, further cause the at least one processor of the portable device to:
take the image of the 3D environment with at least one camera; and identify a plurality of groups of pixels as representatives of features of the image, wherein the one or more feature descriptors are computed as representing the plurality of groups of pixels.

8 . A system for providing shared location-based content to at least one portable device capable of rendering virtual content in a 3D environment, the system comprising:
one or more processors; and at least one computer-readable medium comprising computer executable instructions that, when executed by the one or more processors, cause the one or more processors to:
receive, from a portable electronic device, one or more feature descriptors each representing a respective feature of an image of the 3D environment captured by the portable electronic device;
calculate a second frame descriptor for the image based on the received one or more feature descriptors; and
localize the portable electronic device to a shared coordinate system based on the second frame descriptor,
wherein the second frame descriptor has a second resolution greater than a first resolution of a first frame descriptor calculated at the at least one portable device.

9 . The system of claim 8 , wherein:
the at least one computer-readable medium further comprises a plurality of stored maps of the 3D environment; and at least one map of the plurality of stored maps of the 3D environment is associated with at least one frame descriptor with a resolution greater than 512 bits.

10 . The system of claim 9 , wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to localize the portable electronic device based on comparing compare the second frame descriptor for the image with the at least one frame descriptor associated with the at least one map of the plurality of stored maps of the 3D environment.

11 . The system of claim 10 , wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to localize the portable electronic device by selecting one or more maps from the plurality of stored maps associated with the shared coordinate system, wherein the selecting is based on comparing the second frame descriptor for the image with the at least one frame descriptor associated with the at least one map of the plurality of stored maps.

12 . The system of claim 11 , wherein the computer executable instructions, when executed by the one or more processors, further cause the one or more processors to send, to the portable electronic device, the selected one or more maps.

13 . The system of claim 10 , wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to localize the portable electronic device by determining whether a location of the portable electronic device corresponds to a stored map from the plurality of stored maps of the 3 D environment, wherein the determining is based on comparing the second frame descriptor for the image with the at least one frame descriptor associated with the at least one map of the plurality of stored maps.

14 . The system of claim 13 , wherein the computer executable instructions, when executed by the one or more processors, further cause the one or more processors to:
receive a tracking map from the portable electronic device; merge the tracking map with the stored map, to produce a merged map comprising location information based on location information of the stored map and the tracking map; and store the merged map in the at least one computer-readable medium.

15 . The system of claim 8 , wherein the portable electronic device is selected from a group comprising:
a wearable device comprising a head mounted display comprising a plurality of cameras

Assignees

• Magic Leap Inc

Inventors

• Joseph Elad
• Braun Gal
• Shahrokni Ali

Classifications

• G06V10/40

  Extraction of image or video features · CPC title

• G06V40/18

  Eye characteristics, e.g. of the iris · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title