Cross reality system

Patent metadata

Field	Value
Publication number	US-11227435-B2
Application number	US-201916593745-A
Country	US
Kind code	B2
Filing date	Oct 4, 2019
Priority date	Aug 13, 2018
Publication date	Jan 18, 2022
Grant date	Jan 18, 2022

Abstract

A cross reality system that provides an immersive user experience by storing persistent spatial information about the physical world that one or multiple user devices can access to determine position within the physical world and that applications can access to specify the position of virtual objects within the physical world. Persistent spatial information enables users to have a shared virtual, as well as physical, experience when interacting with the cross reality system. Further, persistent spatial information may be used in maps of the physical world, enabling one or multiple devices to access and localize into previously stored maps, reducing the need to map a physical space before using the cross reality system in it. Persistent spatial information may be stored as persistent coordinate frames, which may include a transformation relative to a reference orientation and information derived from images in a location corresponding to the persistent coordinate frame.

First claim

What is claimed:

1. An electronic system, comprising:
one or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information comprising a plurality of images; and
at least one processor configured to execute computer executable instructions to generate a map of at least a portion of the 3D environment based on the plurality of images, wherein the computer executable instructions further comprise instructions for:
identifying a plurality of features in the plurality of images based at least in part on a criteria indicating persistence associated with the identified features;
selecting a plurality of key frames from among the plurality of images based, at least in part, on the plurality of features of the selected key frames;
generating one or more coordinate frames based, at least in part, on the identified features of the selected key frames; and
storing, in association with the map of the 3D environment, the one or more coordinate frames as one or more persistent coordinate frames.

2. The electronic system of claim 1 , wherein:
the one or more sensors comprises a plurality of pixel circuits arranged in a two-dimensional array such that each image of the plurality of images comprises a plurality of pixels; and
each feature corresponds to a plurality of pixels.

3. The electronic system of claim 1 , wherein storing the one or more coordinate frames comprises storing for each of the one or more coordinate frames:
descriptors representative of at least a subset of the features in a selected key frame from which the coordinate frame was generated.

4. The electronic system of claim 1 , wherein storing the one or more coordinate frames comprises storing for each of the one or more coordinate frames:
at least a subset of the features in a selected key frame from which the coordinate frame was generated.

5. The electronic system of claim 1 , wherein storing the one or more coordinate frames comprises storing for each of the one or more coordinate frames:
a transformation between a coordinate frame of the map of the 3D environment and the persistent coordinate frame; and
geographic information indicating a location within the 3D environment of a selected key frame from which the coordinate frame was generated.

6. The electronic system of claim 5 , wherein
the geographic information comprises a WiFi fingerprint of the location.

7. The electronic system of claim 1 , wherein the computer executable instructions comprise instructions for computing feature descriptors for individual features with an artificial neural network.

8. The electronic system of claim 7 , wherein:
the first artificial neural network is a first artificial neural network; and
the computer executable instructions comprise instructions for implementing a second artificial neural network configured to compute a frame descriptor to represent a key frame based, at least in part, on the computed feature descriptors for the identified features in the key frame.

9. The electronic system of claim 1 , wherein:
the computer executable instructions further comprise:
an application programming interface configured to provide to an application, executing on the portable electronic system, information characterizing a persistent coordinate frame of the one or more persistent coordinate frames;
instructions for refining the map of the 3D environment based on a second plurality of images;
adjusting one or more of the persistent coordinate frames based, at least in part, on the second plurality of images; and
instructions for providing through the application programming interface, notification of the adjusted persistent coordinate frames.

10. The electronic system of claim 9 , wherein:
adjusting the one or more persistent coordinate frames comprises adjusting a translation and rotation of the one or more persistent coordinate frames relative to an origin of the map of the 3D environment.

11. The electronic system of claim 10 , wherein:
the electronic system comprises a wearable device and the one or more sensors are mounted on the wearable device;
the map is a tracking map computed on the wearable device, and
the origin of the map is determined based on a location where the device is powered on.

12. An electronic system, comprising:
one or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information comprising a plurality of images; and
at least one processor configured to execute computer executable instructions to generate a map of at least a portion of the 3D environment based on the plurality of images, wherein the computer executable instructions further comprise instructions for:
identifying a plurality of features in the plurality of images;
selecting a plurality of key frames from among the plurality of images based, at least in part, on the plurality of features of the selected key frames;
generating one or more coordinate frames based, at least in part, on the identified features of the selected key frames; and
storing, in association with the map of the 3D environment, the one or more coordinate frames as one or more persistent coordinate frames, wherein:

identifying the plurality of features in the plurality of images comprises selecting as the identified features a number, less than a predetermined maximum, of groups of the pixels based on a measure of similarity to groups of pixels depicting portions of persistent objects.

13. The electronic system of claim 12 , wherein:
the one or more sensors comprises a plurality of pixel circuits arranged in a two-dimensional array such that each image of the plurality of images comprises a plurality of pixels; and
each feature corresponds to a plurality of pixels.

14. The electronic system of claim 12 , wherein storing the one or more coordinate frames comprises storing for each of the one or more coordinate frames:
descriptors representative of at least a subset of the features in a selected key frame from which the coordinate frame was generated.

15. The electronic system of claim 12 , wherein storing the one or more coordinate frames comprises storing for each of the one or more coordinate frames:
at least a subset of the features in a selected key frame from which the coordinate frame was generated.

16. The electronic system of claim 12 , wherein storing the one or more coordinate frames comprises storing for each of the one or more coordinate frames:
a transformation between a coordinate frame of the map of the 3D environment and the persistent coordinate frame; and
geographic information indicating a location within the 3D environment of a selected key frame from which the coordinate frame was generated.

17. The electronic system of claim 12 , wherein the computer executable instructions comprise instructions for computing feature descriptors for individual features with an artificial neural network.

18. The electronic system of claim 17 , wherein:
the first artificial neural network is a first artificial neural network; and
the computer executable instructions comprise instructions for implementing a second artificial neural network configured to compute a frame descriptor to represent a key frame based, at least in part, on the computed feature descriptors for the identified features in the key frame.

19. The electronic system of claim 12 , wherein:
the computer executable instructions further comprise:
an application programming interface configured to provide

Assignees

• Magic Leap Inc

Inventors

• Mohan Anush
• Torres Rafael Domingos
• Olshansky Daniel
• Miller Samuel A
• Tajik Jehangir
• Holder Joel David
• Miranda Jeremy Dwayne
• Taylor Robert Blake
• Swaminathan Ashwin
• Agarwal Lomesh
• Barot Hiral Honar
• Suzuki Helder Toshiro
• Shahrokni Ali
• Guendelman Eran
• SINGHAL Prateek
• Zhao Xuan
• Choudhary Siddharth
• Kramer Nicholas Atkinson
• Tossell Kenneth William
• Moore Christian Ivan Robert

Classifications

• H04L67/12

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title

• G06T17/05 Primary

  Geographic models · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• H04L67/01

  Protocols · CPC title

• G06T2207/20216

  Image averaging · CPC title