Localization determination for mixed reality systems

Patent metadata

Field	Value
Publication number	US-11206507-B2
Application number	US-202016945613-A
Country	US
Kind code	B2
Filing date	Jul 31, 2020
Priority date	Jan 23, 2017
Publication date	Dec 21, 2021
Grant date	Dec 21, 2021

Abstract

To enable shared user experiences using augmented reality systems, shared reference points must be provided to have consistent placement (position and orientation) of virtual objects. Furthermore, the position and orientation (pose) of the users must be determinable with respect to the same shared reference points. However, without highly sensitive and expensive global positioning system (GPS) devices, pose information can be difficult to determine to a reasonable level of accuracy. Therefore, what is provided is an alternative approach to determining pose information for augmented reality systems, which can be used to perform location based content acquisition and sharing. Further, what is provided is an alternative approach to determining pose information for augmented reality systems that uses information from already existing GPS devices.

First claim

What is claimed:

1. A method for determining position and orientation information of an augmented reality display device, comprising:
identifying a map area associated with a current location of the augmented reality display device;
determining an execution order of multiple localization modules for localizing the augmented reality display device based at least in part upon whether the current location associated with the map area is a determined or identified area to the augmented reality display device;
in response to a determination that the current location associated with the map area includes an undetermined or unidentified area, determining a matching keyframe that is used to recognize the undetermined or unidentified area for the augmented reality display device, determining the matching keyframe comprising:
triggering a first execution of a first localization module that compares a keyframe captured by the augmented reality display device to a plurality of known keyframes associated with the map area; and
selecting a known keyframe from the plurality of known keyframes as the matching keyframe based at least in part on a respective similarity score determined for each of the plurality of known keyframes; and

after the matching keyframe is determined, determining a pose for the augmented reality display device at least by triggering a second execution of a second localization module that determines a first matching keyframe, wherein the second execution comprises:
determining one or more features in the keyframe captured by the augmented reality display device;
comparing the one or more features in the keyframe to one or more corresponding features in at least one keyframe of the plurality of known keyframes; and
determining a first keyframe from the plurality of known keyframes based at least in part a result of comparing the one or more features.

2. The method of claim 1 , wherein identifying the map area associated with the current location comprises:
receiving metadata, the metadata collected at the current location of the augmented reality display device; and
analyzing the metadata to determine the map area, wherein the metadata corresponds to information pertaining to one or more communications networks.

3. The method of claim 1 , wherein the pose for the augmented reality display device is determined with respect to a shared map area origin point, and determining the pose comprises:
determining a first pose relative to the matching keyframe using at least the keyframe captured by the augmented reality display device and the matching keyframe by performing a keyframe analysis; and
converting the first pose relative to the matching keyframe into the pose relative to the shared map area origin point using at least the first pose relative to the matching keyframe.

4. The method of claim 3 , wherein converting the first pose relative to the matching keyframe into the pose relative to the shared map area origin point comprises performing first and second transformations on the pose relative to the matching keyframe, the first transformation is from the shared map area origin point to the matching keyframe, and the second transformation is from the matching keyframe to the keyframe captured by the augmented reality display device.

5. The method of claim 1 , wherein identifying the matching keyframe associated with the map area further comprises:
determining features of the keyframe captured by the augmented reality device, wherein
comparing the keyframe captured by the augmented reality device to the plurality of known keyframes associated with the map area comprises comparing the features of the keyframe to corresponding features of the plurality of known keyframes, and
determining a similarity score for at least some of the plurality of known keyframes, wherein determining the similarity score comprises analyzing the features of the keyframe with respect to the corresponding features of the plurality of known keyframes.

6. The method of claim 1 , further comprising transmitting location-based content in response to at least one of identifying the map area, identifying the matching keyframe, or determining the pose, wherein the location-based content corresponds to data for one or more objects or one or more applications associated with at least one of the map area, the matching keyframe, or the pose.

7. The method of claim 1 , further comprising transmitting user data for performing content sharing in response to at least one of identifying the map area, determining the matching keyframe, or determining the pose, wherein the content sharing comprises one-way or two-way exchange of content between devices over a local communications network.

8. The method of claim 1 , further comprising transmitting map area data in response to identifying the map area; and updating the plurality of known keyframes.

9. The method of claim 1 , further comprising:
after the first matching keyframe is determined, determining the pose for the augmented reality display device at least further by triggering a third execution of a third localization module, the third execution comprising:
determining a relative pose with respect to the first matching keyframe;
determining a first position of the augmented reality display device with respect to an origin of the map area at least by chaining a first transform from the origin to the keyframe and a second transform from the keyframe to the augmented reality display device;
translating a relative position with respect to the first matching keyframe to a first relative position with respect to the origin of the map area based at least in part upon the first and the second transforms; and
determining the pose for the augmented reality display device with respect to the origin based at least in part upon the first relative position.

10. The method of claim 9 , wherein the first matching frame is different from the matching keyframe.

11. The method of claim 1 , wherein the keyframe captured by the augmented reality display device is received from an imaging device coupled to the augmented reality display device, and the imaging device has a known position relative to a user of the augmented reality display device.

12. The method of claim 1 , the first execution of the first localization module comprising:
receiving a first wireless signal corresponding to a first known location and a first known boundary;
identifying a first pair of distances based at least in part upon first metadata associated with the first wireless signal; and
determining a first subset of map areas in a map in which the augmented reality display device is located based at least in part upon the first pair of distances, wherein the plurality of known keyframes associated with the map area is stored as parts of a map area data set.

13. The method of claim 12 , further comprising:
receiving a second wireless signal corresponding to a second known location and a second known boundary;
identifying a second pair of distances based at least in part upon second metadata associated with the second wireless signal; and
determining a second subset from the first subset in which the augmented reality display device is located based at least in part upon the second pair of distances.

14. The method of claim 13 , further comprising:
determining an overlap between the first subset and the second subset; and
determining the position and the orientation information of the augmented reality display device based at least in part upon the overlap.

15. A method for determining position and orientation

Assignees

• Magic Leap Inc

Inventors

• Reinhardt Benjamin Zaaron
• Kho Samuel Phoa

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• H04W4/026 Primary

  using orientation information, e.g. compass · CPC title

• G06F16/907

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title

• G06F16/29

  Geographical information databases · CPC title