Localization determination for mixed reality systems

Patent metadata

Field	Value
Publication number	US-2020367018-A1
Application number	US-202016945613-A
Country	US
Kind code	A1
Filing date	Jul 31, 2020
Priority date	Jan 23, 2017
Publication date	Nov 19, 2020
Grant date	—

Abstract

To enable shared user experiences using augmented reality systems, shared reference points must be provided to have consistent placement (position and orientation) of virtual objects. Furthermore, the position and orientation (pose) of the users must be determinable with respect to the same shared reference points. However, without highly sensitive and expensive global positioning system (GPS) devices, pose information can be difficult to determine to a reasonable level of accuracy. Therefore, what is provided is an alternative approach to determining pose information for augmented reality systems, which can be used to perform location based content acquisition and sharing. Further, what is provided is an alternative approach to determining pose information for augmented reality systems that uses information from already existing GPS devices.

First claim

What is claimed:

1 . A method for determining position and orientation information of an augmented reality display device, comprising:
identifying a map area associated with a current location of the augmented reality display device; identifying a matching keyframe associated with the map area by receiving a keyframe at the augmented reality display device,
comparing the received keyframe to a plurality of known keyframes associated with the map area, the plurality of known keyframes comprising previously captured keyframes having known poses, the known poses comprising at least information representing position and orientation for the respective keyframes,
determining a similarity score for at least some of the plurality of known keyframes by analyzing the received keyframe with respect to the plurality of known keyframes, and
selecting a keyframe of the plurality of known keyframes as the matching keyframe based on at least the similarity score; and
determining a pose for the augmented reality display device with respect to a shared map area origin point using at least the matching keyframe.

2 . The method of claim 1 , wherein identifying the map area associated with the current location comprises:
receiving metadata, the metadata collected at the current location of the augmented reality display device; and analyzing the metadata to determine the map area, wherein the metadata corresponds to information pertaining to one or more communications networks.

3 . The method of claim 1 , wherein determining a pose for the augmented reality display device with respect to the shared map area origin point comprises:
determining a pose relative to the matching keyframe using the received keyframe and the matching keyframe by performing keyframe analysis; and converting the pose relative to the matching keyframe into a pose relative to the shared map area origin point using at least the pose relative to the matching keyframe.

4 . The method of claim 3 , wherein converting the pose relative to the matching keyframe into a pose relative to the shared map area origin point using comprises performing first and second transformations on the pose relative to the matching keyframe.

5 . The method of claim 4 , wherein the first transformation is from the shared map area origin point to the matching keyframe, and
wherein the second transformation is from the matching keyframe to the received keyframe.

6 . The method of claim 1 , wherein identifying the matching keyframe associated with the map area further comprises:
determining features of the received keyframe; wherein comparing the received keyframe to the plurality of known keyframes associated with the map area comprises comparing the features of the received keyframe to features of the plurality of known keyframes, and wherein determining a similarity score for at least some of the plurality of known keyframes comprises analyzing the features of the received keyframe with respect to the features of the plurality of known keyframes.

7 . The method of claim 1 , further comprising transmitting location based content in response to at least one of identifying the map area, identifying the matching keyframe, or determining the pose, wherein location based content corresponds to data for one or more objects or one or more applications associated with at least one of the map area, the matching keyframe, or the pose.

8 . The method of claim 1 , further comprising transmitting user data for performing content sharing in response to at least one of identifying the map area, identifying the matching keyframe, or determining the pose, wherein content sharing comprises one-way or two-way exchange of content between devices over a local communications network.

9 . The method of claim 1 , further comprising transmitting map area data in response to identifying the map area.

10 . The method of claim 1 , further comprising updating the plurality of known keyframes.

11 . The method of claim 1 , wherein the received keyframe at the augmented reality display device is received from an imaging device coupled to the augmented reality display device.

12 . The method of claim 11 , wherein the imaging device has a known position relative to a user of the augmented reality display device.

13 . The method of claim 1 , wherein the plurality of known keyframes associated with the map area is stored as parts of a map area data set.

14 . The method of claim 1 , wherein the plurality of known keyframes associated with the map area is stored as a keyframe/features database, wherein individual keyframes and associated features are identified with one or more identifiers.

15 . The method of claim 1 , wherein analyzing the received keyframe with respect to the plurality of known keyframes comprises comparing a feature in the received keyframe with respective features in the plurality of known keyframes.

16 . A method for determining position and orientation information of an augmented reality display device, comprising:
identifying a map area associated with a current location of the augmented reality display device; identifying a first matching keyframe associated with the map area by
receiving a first plurality of keyframes at the augmented reality display device,
comparing a first keyframe of the first plurality of keyframes to a second plurality of known keyframes associated with the map area, the second plurality of known keyframes comprising previously captured keyframes having known poses, the known poses comprising at least information representing position and orientation for the respective keyframes,
determining a first similarity score for at least some of the second plurality of known keyframes by analyzing the first keyframe with respect to the second plurality of known keyframes, and
selecting a keyframe of the second plurality of known keyframes as the first matching keyframe based on at least the first similarity score; and
determining a first pose for the augmented reality display device with respect to a shared map area origin point using at least the first matching keyframe.

17 . The method of claim 16 , further comprising:
identifying a second matching keyframe associated with the map area by
comparing a second keyframe of the first plurality of keyframes to the second plurality of known keyframes associated with the map area,
determining a second similarity score for at least some of the second plurality of known keyframes by analyzing the second keyframe with respect to the second plurality of known keyframes, and
selecting a keyframe of the second plurality of known keyframes as the second matching keyframe based on at least the second similarity score; and
refining the first pose for the augmented reality display device with respect to the shared map area origin point using at least the second matching keyframe.

18 . The method of claim 16 , wherein identifying the map area associated with the current location comprises:
receiving metadata, the metadata collected at the current location of the augmented reality display device; and analyzing the metadata to determine the map area, wherein the metadata corresponds to information pertaining to one or more communications networks.

19 . The method of claim 16 , wherein determining a pose for the augmented reality display device with respect to the shared map area origin point comprises:
determining a pose relative to the first matching keyframe using the first received keyframe and the fir

Assignees

• Magic Leap Inc

Inventors

• Reinhardt Benjamin Zaaron
• Kho Samuel Phoa

Classifications

• H04W4/027

  using movement velocity, acceleration information · CPC title

• H04W4/026 Primary

  using orientation information, e.g. compass · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06F16/907

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title