Enhanced pose determination for display device

Patent metadata

Field	Value
Publication number	US-11501529-B2
Application number	US-202117193568-A
Country	US
Kind code	B2
Filing date	Mar 5, 2021
Priority date	Dec 15, 2017
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

To determine the head pose of a user, a head-mounted display system having an imaging device can obtain a current image of a real-world environment, with points corresponding to salient points which will be used to determine the head pose. The salient points are patch-based and include: a first salient point being projected onto the current image from a previous image, and with a second salient point included in the current image being extracted from the current image. Each salient point is subsequently matched with real-world points based on descriptor-based map information indicating locations of salient points in the real-world environment. The orientation of the imaging devices is determined based on the matching and based on the relative positions of the salient points in the view captured in the current image. The orientation may be used to extrapolate the head pose of the wearer of the head-mounted display system.

First claim

What is claimed is:

1. A system comprising:
one or more outwardly-facing imaging devices wearable by a user and configured to obtain images of a real-world environment in a vicinity of the user;
one or more processors; and
one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
matching a portion of a current image of the real-world environment with a patch stored by the system, the patch being associated with a first salient point being tracked by the system, the first salient point being included in a prior image of the real-world environment, wherein the matching is usable to identify the first salient point in the current image, and wherein the first salient point represents a first feature of the real-world environment;
accessing respective descriptors for the first salient point and a second salient point identified in the current image, wherein the second salient point represents a second feature of the real-world environment, and wherein the system stores a descriptor-based map of the real- world environment indicating real-world locations associated with the first feature and the second feature; and
determining a pose associated with the system, the pose being based on the accessed descriptors and the descriptor-based map, and the pose indicating at least an orientation of the one or more outwardly-facing imaging devices in the real-world environment, and being indicative of a head pose of the user of the system.

2. The system of claim 1 , wherein the portion of the current image is matched with the patch stored by the system based on minimizing a cost function.

3. The system of claim 1 , wherein the portion of the current image is matched with the patch using information from an inertial measurement unit of the system.

4. The system of claim 1 , wherein the descriptors are generated based on respective image pixels associated with respective locations of the first salient point and the second salient point in the current image.

5. The system of claim 1 , wherein the descriptors are generated based on respective image areas associated with respective locations of the first salient point and the second salient point in the current image.

6. The system of claim 1 , wherein the descriptor-based map includes a plurality of descriptors associated with a plurality of features, the features including the first feature and the second feature.

7. The system of claim 6 , wherein the operations further comprise matching a subset of the plurality of descriptors with the accessed descriptors to identify at least a first descriptor and a second descriptor which match with the accessed descriptors.

8. The system of claim 7 , wherein the descriptor-based map includes three-dimensional coordinates of salient points associated with the plurality of descriptors, and wherein the pose is based on comparing:
(1) the three-dimensional locations associated with the first descriptor and the second descriptor; and
(2) two-dimensional locations of the first salient point and the second salient point in the current image.

9. The system of claim 1 , wherein the second salient point is extracted from the current image, and wherein extracting comprises:
determining that an image area of the current image has less than a threshold number of salient points being tracked by the system; and
extracting one or more additional salient points from the image area, the extracted salient points including the second salient point.

10. The system of claim 1 , wherein matching the portion of the current image with the patch comprises projecting the patch onto the current image and refining a location associated with the patch.

11. The system of claim 1 , wherein the operations further comprise:
projecting salient points included in the descriptor-based map onto the current image, wherein the projection is based on one or more of an inertial measurement unit, an extended kalman filter, or visual-inertial odometry.

12. A method implemented by a head-mounted system including one or more outwardly-facing imaging devices wearable by a user and configured to obtain images of a real-world environment in a vicinity of the user, the method comprising:
matching a portion of a current image of the real-world environment with a patch stored by the system, the patch being associated with a first salient point being tracked by the system, the first salient point being included in a prior image of the real-world environment, wherein the matching is usable to identify the first salient point in the current image, and wherein the first salient point represents a first feature of the real-world environment;
accessing respective descriptors for the first salient point and a second salient point identified in the current image, wherein the second salient point represents a second feature of the real-world environment, and wherein the system stores a descriptor-based map of the real-world environment indicating real-world locations associated with the first feature and the second feature; and
determining a pose associated with the system, the pose being based on the descriptors and the descriptor-based map, and the pose indicating at least an orientation of the one or more outwardly-facing imaging devices in the real-world environment, and being indicative of a head pose of the user of the head-mounted system.

13. The method of claim 12 , wherein the portion of the current image is matched with the patch stored by the system based on minimizing a cost function.

14. The method of claim 12 , wherein the portion of the current image is matched with the patch using information from an inertial measurement unit of the system.

15. The method of claim 12 , wherein the descriptors are generated based on respective image pixels associated with respective locations of the first salient point and second salient point in the current image.

16. The method of claim 12 , wherein the descriptors are generated based on respective image areas associated with respective locations of the first salient point and second salient point in the current image.

17. The method of claim 12 , wherein the descriptor-based map includes a plurality of descriptors associated with a plurality of features, the features including the first feature and the second feature.

18. The method of claim 17 , wherein the method further comprises matching a subset of the plurality of descriptors with the accessed descriptors to identify at least a first descriptor and a second descriptor which match with the accessed descriptors.

19. The method of claim 18 , wherein the descriptor-based map includes three-dimensional coordinates of salient points associated with the plurality of descriptors, and wherein the pose is based on comparing:
(1) the three-dimensional locations associated with the first descriptor and the second descriptor; and
(2) two-dimensional locations of the first salient point and the second salient point in the current image.

20. The method of claim 12 , wherein the second salient point is extracted from the current image, and wherein extracting comprises:
determining that an image area of the current image has less than a threshold number of salient points being tracked by the system; and
extracting one or more additional salient points from the image area, the extracted salient points including the second salient point.

21. The method of claim 12 , wh

Assignees

• Magic Leap Inc

Inventors

• Zahnert Martin Georg
• Faro Joao Antonio Pereira
• Velasquez Miguel Andres Granados
• Kasper Dominik Michael
• Swaminathan Ashwin
• Mohan Anush
• SINGHAL Prateek

Classifications

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F1/163 Primary

  Wearable computers, e.g. on a belt · CPC title

• G06V20/20 Primary

  in augmented reality scenes · CPC title