Cross reality system with map processing using multi-resolution frame descriptors

Patent metadata

Field	Value
Publication number	US-11562525-B2
Application number	US-202117174193-A
Country	US
Kind code	B2
Filing date	Feb 11, 2021
Priority date	Feb 13, 2020
Publication date	Jan 24, 2023
Grant date	Jan 24, 2023

Abstract

A distributed, cross reality system efficiently and accurately compares location information that includes image frames. Each of the frames may be represented as a numeric descriptor that enables identification of frames with similar content. The resolution of the descriptors may vary for different computing devices in the distributed system based on degree of ambiguity in image comparisons and/or computing resources for the device. A descriptor computed for a cloud-based component operating on maps of large areas that can result in ambiguous identification of multiple image frames may use high resolution descriptors. High resolution descriptors reduce computationally intensive disambiguation processing. A portable device, which is more likely to operate on smaller maps and less likely to have the computational resources to compute a high resolution descriptor, may use a lower resolution descriptor.

First claim

What is claimed is:

1. A networked resource within a distributed computing environment for providing shared location-based content to a plurality of portable electronic devices capable of rendering virtual content in a three-dimensional (3D) environment, the networked resource comprising:
one or more processors; and
at least one computer readable medium comprising:
a plurality of stored maps of the 3D environment; and
computer executable instructions that, when executed by the one or more processors, cause the networked resource to:
receive, from a portable electronic device, information regarding a plurality of features detected in an image captured by the portable electronic device; and
compute a frame descriptor for the image that is representative of the plurality of features detected in the image, and wherein the computed frame descriptor has a resolution greater than 512 bits.

2. The networked resource of claim 1 , wherein:
at least one of the plurality of stored maps of the 3D environment is associated with at least one frame descriptor with a resolution greater than 512 bits.

3. The networked resource of claim 2 , wherein:
the computer executable instructions, when executed by the one or more processors, further cause the networked resource to:
compare the computed frame descriptor for the image with the at least one frame descriptor associated with the at least one of the plurality of stored maps of the 3D environment.

4. The networked resource of claim 3 , wherein:
the computer executable instructions, when executed by one or more processors, further cause the networked resource to:
based on comparing the computed frame descriptor for the image with the at least one frame descriptor associated with the at least one of the plurality of stored maps, select one or more maps from the plurality of stored maps to localize the portable electronic device to a shared coordinate system.

5. The networked resource of claim 4 , wherein:
the computer executable instructions, when executed by a processor of the one or more processors, further cause the networked resource to:
send, to the portable electronic device, the selected one or more maps.

6. The networked resource of claim 3 , wherein:
the computer executable instructions, when executed by a processor of the one or more processors, further cause the networked resource to:
based on comparing the computed frame descriptor for the image with the at least one frame descriptor associated with the at least one of the plurality of stored maps, determine whether a location of the portable electronic device corresponds to a stored map from the plurality of stored maps of the 3D environment.

7. The networked resource of claim 6 , wherein:
the computer executable instructions, when executed by a processor of the one or more processors, further cause the networked resource to:
receive a tracking map from the portable electronic device;
merge the tracking map with the stored map, to produce a merged map comprising location information based on location information of the stored map and the tracking map; and
store the merged map in the computer-readable medium.

8. The networked resource of claim 1 , wherein the portable electronic device is selected from a group comprising:
a wearable device comprising a head mounted display comprising a plurality of cameras mounted thereon; and
a portable computing device comprising a camera and a display and configured with computer-executable instructions for rendering virtual content on the display in connection with an image acquired by the camera.

9. The networked resource of claim 1 , wherein the computer executable instructions comprise instructions for implementing a neural network to compute frame descriptors.

10. A method of providing shared location-based content to a plurality of portable electronic devices capable of rendering virtual content in a three-dimensional (3D) environment, the method comprising:
on a portable electronic device:
obtaining one or more images of the 3D environment;
identifying one or more features from the one or more images;
transmitting, to a networked resource, information regarding one or more features identified in the one or more images of the 3D environment; and
computing one or more first frame descriptors of the one or more images based on the one or more features, the first frame descriptors being representative of the one or more features and having a first resolution; and

on the networked resource:
storing a plurality of maps of the 3D environment; and
computing one or more second frame descriptors representing the one or more images based on the information regarding the one or more features,
wherein the one or more second frame descriptors are representative of the one or more features and have a second resolution greater than the first resolution.

11. The method of claim 10 , further comprising:
on the portable electronic device, selecting at least a portion of a local map based on the one or more first frame descriptors; and
on the networked resource, selecting at least a portion of a shared map based on the one or more second frame descriptors.

12. The method of claim 11 , wherein selecting at least the portion of the shared map comprises comparing the one or more second frame descriptors with one or more frame descriptors associated with the plurality of maps of the 3D environment.

13. The method of claim 12 , further comprising, on the networked resource, based on comparing the one or more second frame descriptors with one or more frame descriptors associated with the plurality of maps of the 3D environment, determining one or more maps from the plurality of maps for localizing the portable electronic device to a shared coordinate system.

14. The method of claim 13 , further comprising:
computing, for the determined one or more maps for localizing the portable electronic device, one or more third frame descriptors, wherein the one or more third frame descriptors have the first resolution;
sending to the portable electronic device, from the networked resource, the determined one or more maps for localizing the portable electronic device and the one or more third frame descriptors.

15. The method of claim 11 , further comprising, on the networked resource, based on comparing one or more frame descriptors calculated from information regarding clusters of pixels received from the portable electronic device with one or more frame descriptors associated with the plurality of maps of the 3D environment, determining whether a location of the portable electronic device corresponds to a stored map from the plurality of stored maps of the 3D environment.

16. The method of claim 15 , wherein:
the information regarding the one or more features comprises a tracking map; and
the method further comprises:
merging, at the network resource, the tracking map with the stored map, to produce a merged map comprising location information based on location information of the stored map and the tracking map; and
storing, at the network resource, the merged map in the computer-readable medium in conjunction with the one or more second frame descriptors.

17. The method of claim 10 , wherein the portable electronic device is selected from a group comprising:
a wearable device comprising a head mounted display comprising a plurality of cameras mounted thereon; and
a portable computing device comprising a camera and a display and configured with computer-executable instructions for rendering virtual content o

Assignees

• Magic Leap Inc

Inventors

• Joseph Elad
• Braun Gal
• Shahrokni Ali

Classifications

• H04N5/2257

  Electricity · mapped topic

• G06V10/40

  Extraction of image or video features · CPC title

• G06T15/005 Primary

  General purpose rendering architectures · CPC title

• H04N5/247

  Electricity · mapped topic

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title