Cross reality system

Patent metadata

Field	Value
Publication number	US-11978159-B2
Application number	US-202117542306-A
Country	US
Kind code	B2
Filing date	Dec 3, 2021
Priority date	Aug 13, 2018
Publication date	May 7, 2024
Grant date	May 7, 2024

Abstract

A cross reality system that provides an immersive user experience by storing persistent spatial information about the physical world that one or multiple user devices can access to determine position within the physical world and that applications can access to specify the position of virtual objects within the physical world. Persistent spatial information enables users to have a shared virtual, as well as physical, experience when interacting with the cross reality system. Further, persistent spatial information may be used in maps of the physical world, enabling one or multiple devices to access and localize into previously stored maps, reducing the need to map a physical space before using the cross reality system in it. Persistent spatial information may be stored as persistent coordinate frames, which may include a transformation relative to a reference orientation and information derived from images in a location corresponding to the persistent coordinate frame.

First claim

What is claimed:

1. A method of operating an electronic system to render virtual content in a 3D environment comprising a portable device, the method comprising, with one or more processors:
maintaining on the portable device a local coordinate frame based on output of one or more sensors on the portable device;
obtaining a stored coordinate frame from stored spatial information about the 3D environment;
computing a transformation between the local coordinate frame and the obtained stored coordinate frame;
receiving a specification of a virtual object having a virtual object coordinate frame and a location of the virtual object with respect to the obtained stored coordinate frame; and
rendering the virtual object on a display of the portable device at a location determined, at least in part, based on the computed transformation and the received location of the virtual object.

2. The method of claim 1 , wherein:
obtaining the stored coordinate frame comprises obtaining the coordinate frame through an application programming interface (API).

3. The method of claim 1 , wherein:
the portable device comprises a first portable device comprising a first processor of the one or more processors;
the system further comprises a second portable device comprising a second processor of the one or more processors; and
processors on each of the first and second devices:
obtains a same, stored coordinate frame;
computes a transformation between a respective local coordinate frame of the obtained stored coordinate frame;
receives the specification of the virtual object; and
renders the virtual object on a respective display so as to create a shared perception of the virtual object in the 3D environment for users of each of the first and second devices.

4. The method of claim 3 , wherein each of the first and second devices comprises:
a camera configured to output a plurality of camera images;
a key frame generator configured to transform a plurality camera images to a plurality of key frames;
a persistent pose calculator configured to generate a persistent pose by averaging the plurality of key frames;
a tracking map and persistent pose transformer configured to transform a tracking map to the persistent pose to determine the persistent pose at an origin relative to the tracking map;
a persistent pose and PCF transformer configured to transform the persistent pose to a PCF; and
a map publisher, configured to transmit spatial information, including the PCF, to a server.

5. The method of claim 3 , further comprising:
executing an application to generate the specification of the virtual object and the location of the virtual object with respect to the obtained stored coordinate frame.

6. The method of claim 3 , wherein:
maintaining on the portable device a local coordinate frame comprises, for each of the first and second portable devices:
capturing a plurality of images about the 3D environment from the one or more sensors of the portable device,
computing one or more persistent poses based, at least in part, on the plurality of images, and
generating spatial information about the 3D environment based, at least in part, on the computed one or more persistent poses;
the method further comprises, for each of the first and second portable devices transmitting to a remote server the generated spatial information; and
obtaining the stored coordinate frame comprises receiving the stored coordinate frame from the remote server.

7. The method of claim 6 , wherein computing the one or more persistent poses based, at least in part, on the plurality of images comprises:
extracting one or more features from each of the plurality of images;
generating a descriptor for each of the one or more features;
generating a key frame for each of the plurality of images based, at least in part, on the descriptors; and
generating the one or more persistent poses based, at least in part, on the one or more key frames.

8. The method of claim 7 , wherein generating the one or more persistent poses comprises:
selectively generating a persistent pose based on the portable device traveling a pre-determined distance from a location of other persistent poses.

9. The method of claim 3 , wherein each of the first and second devices comprises:
a download system configured to download the stored coordinate frame from a server.

10. A portable device, comprising:
one or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information comprising a plurality of images; and
at least one processor configured to execute computer executable instructions to generate a map of at least a portion of the 3D environment based on the plurality of images, wherein the computer executable instructions further comprise instructions for:
maintaining on the portable device a local coordinate frame based on output of the one or more sensors;
obtaining a stored coordinate frame from stored spatial information about the 3D environment;
computing a transformation between the local coordinate frame and the obtained stored coordinate frame;
receiving a specification of a virtual object having a virtual object coordinate frame and a location of the virtual object with respect to the obtained stored coordinate frame; and
rendering the virtual object on a display of the portable device at a location determined, at least in part, based on the computed transformation and the received location of the virtual object.

11. The portable device of claim 10 , wherein maintaining on the portable device a local coordinate frame comprises:
computing one or more persistent poses based, at least in part, on the plurality of images, and
generating spatial information about the 3D environment based, at least in part, on the computed one or more persistent poses.

12. The portable device of claim 11 , wherein:
the computer executable instructions further comprise instructions for transmitting to a remote server the generated spatial information; and
obtaining the stored coordinate frame comprises receiving the stored coordinate frame from the remote server.

13. The portable device of claim 11 , wherein:
computing the one or more persistent poses based, at least in part, on the plurality of images comprises:
extracting one or more features from each of the plurality of images;
generating a descriptor for each of the one or more features;
generating a key frame for each of the plurality of images based, at least in part, on the descriptors; and
generating the one or more persistent poses based, at least in part, on the one or more key frames.

14. The portable device of claim 13 , wherein generating the one or more persistent poses comprises selectively generating a persistent pose based on the portable device traveling a pre-determined distance from a location of other persistent poses.

Assignees

• Magic Leap Inc

Inventors

• Mohan Anush
• Torres Rafael Domingos
• Olshansky Daniel
• Miller Samuel A
• Tajik Jehangir
• Holder Joel David
• Miranda Jeremy Dwayne
• Taylor Robert Blake
• Swaminathan Ashwin
• Agarwal Lomesh
• Barot Hiral Honar
• Suzuki Helder Toshiro
• Shahrokni Ali
• Guendelman Eran
• SINGHAL Prateek
• Zhao Xuan
• Choudhary Siddharth
• Kramer Nicholas Atkinson
• Tossell Kenneth William
• Moore Christian Ivan Robert

Classifications

• H04L67/12

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title

• G06T17/05 Primary

  Geographic models · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06F16/29

  Geographical information databases · CPC title