Cross reality system supporting multiple device types

Patent metadata

Field	Value
Publication number	US-12293471-B2
Application number	US-202217571172-A
Country	US
Kind code	B2
Filing date	Jan 7, 2022
Priority date	Oct 15, 2019
Publication date	May 6, 2025
Grant date	May 6, 2025

Abstract

A cross reality system enables any of multiple types of devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. Devices of any type, with native hardware and software configured for augmented reality operations may be configured to work with the cross reality system by incorporating components that interface between the native AR framework of the device and the cloud-based localization service. These components may present position information about the device in a format recognized by the localization service. Additionally, these components may filter or otherwise process perception data provided by the native AR framework to increase the accuracy of localization.

First claim

What is claimed is:

1. A method of operating a portable electronic device of a first type in a cross reality system in which devices of multiple types interact with a remote localization service via at least one network, the method comprising:
capturing a plurality of images with one or more sensors on the portable electronic device of the first type, wherein the plurality of images comprise pixels depicting a three-dimensional (3D) environment of the portable electronic device of the first type at a time of capture, and the capturing is controlled by a native AR framework of the portable electronic device of the first type;
modifying, by an interface component of the portable electronic device of the first type, the plurality of images to normalize values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type with respect to values of pixels in images captured with electronic devices of a second type such that the values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type are more consistent with the values of pixels in images captured with electronic devices of the second type, wherein the native AR framework of the portable electronic device of the first type is not configured to operate with the remote localization service, and the electronic devices of the second type comprise native AR frameworks configured to operate with the remote localization service; and
causing information related to at least a subset of the plurality of modified images to be transmitted to the remote localization service, wherein the portable electronic device of the first type comprises one or more components configured to communicate with the remote localization service via the at least one network so as to transmit the at least a subset of the plurality of modified images to the remote localization service, and the interface component of the portable electronic device of the first type is configured to format a response from the remote localization service such that the formatted response can be used by the native AR framework of the portable electronic device of the first type to position virtual content on the device.

2. The method of claim 1 , comprising:
obtaining, from the remote localization service, a transformation between a first coordinate frame local to the portable electronic device of the first type and a second coordinate frame shared with the electronic devices of the second type; and
rendering, based on the obtained transformation, a virtual object positioned in a location with respect to the 3D environment such that the virtual object is accessible to both the portable electronic device of the first type and the electronic devices of the second type at the location with respect to the 3D environment, wherein:
the plurality of images captured with the one or more sensors on the portable electronic device of the first type are two-dimensional (2D) images; and
the electronic devices of the second type are configured to capture three-dimensional (3D) images.

3. The method of claim 1 , wherein modifying the plurality of images to normalize the values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type with respect to the values of the pixels in the images captured with the electronic devices of the second type such that the values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type are more consistent with the values of pixels in images captured with electronic devices of the second type comprises:
changing values for a parameter of interest for the pixels of the plurality of images captured by the sensor to approximate the values expected if the plurality of images were captured with the electronic devices of the second type.

4. The method of claim 1 , wherein modifying the plurality of images to normalize the values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type with respect to the values of the pixels in the images captured with the electronic devices of the second type such that the values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type are more consistent with the values of pixels in images captured with electronic devices of the second type comprises:
mapping values for a parameter of interest for a first number of pixels of one of the plurality of images to a second number of pixels according to the images captured with the electronic devices of the second type, the second number being different from the first number.

5. The method of claim 4 , wherein:
the first number is less than the second number; and
the first number of pixels are interpolated to the second number of pixels such that the second number of pixels forms an image having values as if the image were captured by the electronic devices of the second type.

6. The method of claim 1 , wherein:
the portable electronic device is a handheld device; and
the second type is a wearable device comprising a head mounted display comprising a plurality of cameras mounted thereon.

7. The method of claim 1 , wherein modifying the plurality of images to normalize the values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type with respect to the values of the pixels in the images captured with the electronic devices of the second type such that the values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type are more consistent with the values of pixels in images captured with electronic devices of the second type comprises:
combining at least two images of the plurality of images to form a single normalized image.

8. A portable electronic device of a first type in a cross reality system in which devices of multiple types interact with a remote localization service via at least one network, the portable electronic device of the first type comprising:
one or more sensors configured to capture a plurality of images; and
at least one processor configured to execute computer executable instructions, wherein the computer executable instructions comprise instructions for performing a method comprising:
capturing the plurality of images with the one or more sensors on the portable electronic device of the first type, wherein the plurality of images comprise pixels depicting a three-dimensional (3D) environment of the portable electronic device of the first type at a time of capture, and the capturing is controlled by a native AR framework of the portable electronic device;
modifying, by an interface component of the portable electronic device of the first type, the plurality of images to normalize values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type with respect to values of pixels in images captured with electronic devices of a second type such that the values of the pixels in the plurality of images captured with the one or more sensors on the portable electronic device of the first type are more consistent with the values of pixels in images captured with electronic devices of the second type, wherein the native AR framework of the portable electronic device of the first type is not configured to operate with the remote localization service, and the electronic devices of the second type comprise nativ

Assignees

• Magic Leap Inc

Inventors

• Zhao Xuan
• Shahrokni Ali
• Olshansky Daniel
• Moore Christian Ivan Robert
• Torres Rafael Domingos
• Holder Joel David

Classifications

• G06T5/70

  Denoising; Smoothing · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title