Cross reality system supporting multiple device types

Patent metadata

Field	Value
Publication number	US-11257294-B2
Application number	US-202017071907-A
Country	US
Kind code	B2
Filing date	Oct 15, 2020
Priority date	Oct 15, 2019
Publication date	Feb 22, 2022
Grant date	Feb 22, 2022

Abstract

A cross reality system enables any of multiple types of devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. Devices of any type, with native hardware and software configured for augmented reality operations may be configured to work with the cross reality system by incorporating components that interface between the native AR framework of the device and the cloud-based localization service. These components may present position information about the device in a format recognized by the localization service. Additionally, these components may filter or otherwise process perception data provided by the native AR framework to increase the accuracy of localization.

First claim

What is claimed is:

1. A method of operating a portable electronic device of a first type in a cross reality system in which devices of multiple types interact with a remote localization service via at least one network, the portable electronic device being configured to capture images, the method comprising acts of:
capturing a plurality of images with a sensor on the portable electronic device, wherein the plurality of images comprise pixels depicting a three-dimensional (3D) environment of the portable electronic device at a time of capture;
modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type;
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment;
generating feature descriptors from the selected clusters of pixels; and
causing the feature descriptors to be transmitted to the remote localization service via the at least one network.

2. The method of claim 1 , wherein modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type comprises:
modifying a brightness value for the pixels of the plurality of images captured by the sensor;
applying a gamma correction to the plurality of images; and/or
applying a denoise operation to the plurality of images.

3. The method of claim 2 , wherein selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment further comprises applying a lens shading correction operation to the plurality of images.

4. The method of claim 1 , wherein:
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment comprises selecting clusters of pixels in the subset of the plurality of modified images, wherein the subset is formed, at least in part by filtering out images of the plurality of modified images based on motion blur detected in the images.

5. The method of claim 1 , wherein:
the portable electronic device is a handheld device; and
the second type is a wearable device comprising a head mounted display comprising a plurality of cameras mounted thereon.

6. The method of claim 1 , wherein generating feature descriptors from the selected clusters of pixels comprises generating location information associated with the feature descriptors.

7. The method of claim 1 , further comprising an act of:
obtaining from the remote localization service a transform between a stored coordinate frame of stored spatial information about the 3D environment and a local coordinate frame.

8. A portable electronic device of a first type in a cross reality system in which devices of multiple types interact with a remote localization service via at least one network, the portable electronic device being configured to capture images, comprising:
a sensor configured to capture a plurality of images;
at least one processor configured to execute computer executable instructions, wherein the computer executable instructions comprise instructions for:
capturing the plurality of images with the sensor on the portable electronic device, wherein the plurality of images comprise pixels depicting a three-dimensional (3D) environment of the portable electronic device at a time of capture;
modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type;
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment;
generating feature descriptors from the selected clusters of pixels; and
causing the feature descriptors to be transmitted to the remote localization service via the at least one network.

9. The portable electronic device of claim 8 , wherein modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type comprises:
modifying a brightness value for the pixels of the plurality of images captured by the sensor;
applying a gamma correction to the plurality of images; and/or
applying a denoise operation to the plurality of images.

10. The portable electronic device of claim 9 , wherein selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment further comprises applying a lens shading correction operation to the plurality of images.

11. The portable electronic device of claim 8 , wherein the portable electronic device is a smartphone and the sensor is a camera on the smartphone, and wherein:
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment comprises selecting clusters of pixels in the subset of the plurality of modified images, wherein the subset is formed, at least in part by filtering out images of the plurality of modified images based on motion blur detected in the images.

12. The portable electronic device of claim 8 , wherein:
the portable electronic device is a handheld device; and
the second type is a wearable device comprising a head mounted display comprising a plurality of cameras mounted thereon.

13. The portable electronic device of claim 8 , wherein generating feature descriptors from the selected clusters of pixels comprises generating location information associated with the feature descriptors.

14. The portable electronic device of claim 8 , wherein the computer executable instructions further comprise instructions for:
obtaining from the remote localization service a transform between a stored coordinate frame of stored spatial information about the 3D environment and a local coordinate frame.

15. A non-transitory computer-readable medium storing computer executable instructions configured to, when executed by at least one processor, perform a method for operating a portable electronic device of a first type in a cross reality system in which devices of multiple types interact with a remote localization service via at least one network, the portable electronic device being configured to capture images, the method comprising acts of:
capturing a plurality of images with a sensor on the portable electronic device, wherein the plurality of images comprise pixels depicting a three-dimensional (3D) environment of the portable electronic device at a time of capture;
modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type;
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment;
generating feature descriptors from the selected clusters of pixels; and
causing the feature descriptors to be transmitted to the remote localization service via the at least one network.

16. The non-transitory computer-readable medium of claim 15 , wherein modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type comprises:
modifying a brightness value for the pixe

Assignees

• Magic Leap Inc

Inventors

• Zhao Xuan
• Shahrokni Ali
• Olshansky Daniel
• Moore Christian Ivan Robert
• Torres Rafael Domingos
• Holder Joel David

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T7/73

  using feature-based methods · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T2200/04

  involving 3D image data · CPC title