Localization determination for mixed reality systems

Patent metadata

Field	Value
Publication number	US-10812936-B2
Application number	US-201815877359-A
Country	US
Kind code	B2
Filing date	Jan 22, 2018
Priority date	Jan 23, 2017
Publication date	Oct 20, 2020
Grant date	Oct 20, 2020

Abstract

To enable shared user experiences using augmented reality systems, shared reference points must be provided to have consistent placement (position and orientation) of virtual objects. Furthermore, the position and orientation (pose) of the users must be determinable with respect to the same shared reference points. However, without highly sensitive and expensive global positioning system (GPS) devices, pose information can be difficult to determine to a reasonable level of accuracy. Therefore, what is provided is an alternative approach to determining pose information for augmented reality systems, which can be used to perform location based content acquisition and sharing. Further, what is provided is an alternative approach to determining pose information for augmented reality systems that uses information from already existing GPS devices.

First claim

What is claimed:

1. A method for determining position and orientation information of an augmented reality display device, comprising:
identifying a map area associated with a current location of the augmented reality display device, wherein the map area is identified at least by performing heuristic processing using location information received from one or more devices;
identifying a matching keyframe associated with the map area; and
determining a pose for the augmented reality display device with respect to a shared map area origin point, wherein performing heuristic processing using location information comprises:
identifying location data corresponding to positions of the one or more devices, the location data generated by global positioning systems associated with the one or more devices, and the location data being transmitted wirelessly between the augmented reality display device and respective devices of the one or more devices using respective peer-to-peer communication protocols;
determining a level of accuracy to the location data from at least one of the one or more devices;
estimating a minimum distance and a maximum distance between the augmented reality display device and the at least one of the one or more devices; and
determining at least one estimated region the augmented reality display device is located in using the level of accuracy, the minimum distance, and the maximum distance for at least one of the one or more devices.

2. The method of claim 1 , wherein the minimum distance is predetermined for at least one peer-to-peer communication protocol of the respective peer-to-peer communication protocols.

3. The method of claim 1 , wherein the maximum distance is predetermined for at least one peer-to-peer communication protocol of the respective peer-to-peer communication protocols.

4. The method of claim 1 , wherein the minimum distance is determined based on at least signal strength information for at least one peer-to-peer communication protocol of the respective peer-to-peer communication protocols.

5. The method of claim 1 , wherein the maximum distance is determined based on at least signal strength information for at least one peer-to-peer communication protocol of the respective peer-to-peer communication protocols.

6. The method of claim 1 , wherein the at least one estimated region corresponds to a single map area.

7. The method of claim 1 , wherein a plurality of estimated regions are processed to determine an estimated region of overlap.

8. The method of claim 7 , wherein the estimated region of overlap does not correspond to a single map area.

9. The method of claim 8 , wherein identifying the map area associated with the current location further comprises:
receiving metadata, the metadata collected at the current location of the augmented reality display device; and
analyzing the metadata to determine a matching map area, wherein the metadata corresponds to information pertaining to one or more communication networks.

10. The method of claim 1 , wherein identifying the matching keyframe associated with the map area comprises:
receiving a first plurality of keyframes;
capturing a keyframe of the first plurality of keyframes;
comparing the captured keyframe to a second plurality of keyframes associated with the map area, the second plurality of keyframes comprising previously captured keyframes having known poses, poses comprising at least information representing position and orientation for the respective keyframes;
determining a similarity score for at least some of the second plurality of keyframes by analyzing the captured keyframe with respect to the second plurality of keyframes; and
selecting a keyframe of the second plurality of keyframes as the matching keyframe based on at least the similarity score.

11. The method of claim 10 , wherein determining a pose for the augmented reality display device with respect to the shared map area origin point comprises:
determining a pose relative to the matching keyframe using the captured keyframe and the matching keyframe by performing keyframe analysis; and
converting the pose relative to the matching keyframe into a pose relative to the shared map area origin point using at least the pose corresponding to the matching keyframe.

12. The method of claim 10 , wherein identifying the matching keyframe associated with the map area further comprises:
determining features of the captured keyframe;
wherein comparing the captured keyframe to the second plurality of keyframes associated with the map area comprises comparing the features of the captured keyframe to features of the second plurality of keyframes, and
wherein determining a similarity score for at least some of the second plurality of keyframes comprises analyzing the features of the captured keyframes with respect to the features of the second plurality of keyframes.

Assignees

• Magic Leap Inc

Inventors

• Reinhardt Benjamin Zaaron
• Kho Samuel Phoa

Classifications

• H04W4/027

  using movement velocity, acceleration information · CPC title

• H04W4/026 Primary

  using orientation information, e.g. compass · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T7/74

  involving reference images or patches · CPC title

• G06F16/907

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title