Methods and apparatuses for determining and/or evaluating localizing maps of image display devices

Patent metadata

Field	Value
Publication number	US-12079382-B2
Application number	US-202318314289-A
Country	US
Kind code	B2
Filing date	May 9, 2023
Priority date	Jul 24, 2018
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

An apparatus configured to be worn on a head of a user, includes: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit configured to determine a map based at least in part on output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; wherein the processing unit of the apparatus is also configured to obtain a metric indicating a likelihood of success to localize the user using the map, and wherein the processing unit is configured to obtain the metric by computing the metric or by receiving the metric.

First claim

What is claimed is:

1. A method performed by an apparatus that is configured to be worn on a head of a user, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, the method comprising: obtaining, by the processing unit, output from the camera system; determining a map by the processing unit based at least in part on the output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; and determining, by the processing unit, a score for an image obtained from the camera system, the score indicating how well the image is constrained with respect to a map segment for forming the map, wherein the image is well-constrained when the image forms a well-connected part of the map.

2. The method of claim 1 , wherein the score is determined based on a number of reference points detected in the image.

3. The method of claim 1 , further comprising performing, by the processing unit, data sanitization based on the score.

4. The method of claim 3 , wherein the act of performing the data sanitization comprises removing a constraint of the image, or removing the image.

5. The method of claim 4 , wherein the act of performing the data sanitization comprises removing the constraint of the image, or removing the image, when the score is below a threshold.

6. The method of claim 3 , wherein the sanitization is performed to remove or to disregard data that would otherwise provide an undesirable contribution for the map if the data is used to determine the map.

7. The method of claim 1 , wherein camera system comprises a plurality of cameras, wherein the data comprises a set of images generated by the respective cameras, and wherein the sanitization is performed to remove or disregard the set of images.

8. The method of claim 1 , further comprising performing a bundle adjustment to adjust one or more rays associated with one or more images from the camera system, wherein the bundle adjustment is performed after the sanitization is performed, wherein the image for which the score is determined is one of the one or more images, or is different from the one or more images, wherein the bundle adjustment is performed as a part of a global optimization.

9. The method of claim 8 , further comprising performing, by the processing unit, the global optimization based on the one or more images from the camera system and three-dimensional reference points, wherein the global optimization is performed also based on a relative orientation between cameras of the camera system.

10. The method of claim 1 , wherein the map is determined by: determining multiple map segments, wherein the multiple map segment comprise the map segment; and connecting the map segments, wherein the portion of the map comprises one of the map segments.

11. The method of claim 1 , wherein the camera system is configured to provide additional images, the additional images generated by the camera system before the image for which the score is determined is generated, wherein the act of determining the map comprises determining a first map segment of the map segments by linking the additional images, and wherein the additional images are generated in sequence by the camera system.

12. The method of claim 11 , further comprising: determining respective scores of the additional images; and starting, by the processing unit, a second map segment of the map segments in response to the score of the image from the camera system meeting a criterion, wherein the second map segment is started when the score indicates that the image has a degree of constraint with respect to the first map segment that is below a threshold.

13. The method of claim 1 , wherein the map is determined by: determining multiple map segments, wherein the multiple map segment comprise the map segment; and connecting the map segments; wherein the portion of the map comprises one of the map segments.

14. The method of claim 13 , wherein the camera system is configured to provide additional images, the additional images generated by the camera system before the image for which the score is determined is generated, wherein the act of determining the map comprises determining a first map segment of the map segments by linking the additional images, and wherein the additional images are generated in sequence by the camera system.

15. The method of claim 14 , further comprising: determining respective scores of the additional images; and starting, by the processing unit, a second map segment of the map segments in response to the score of the image from the camera system meeting a criterion, wherein the second map segment is started when the score indicates that the image has a degree of constraint with respect to the first map segment that is below a threshold.

16. The method of claim 1 , further comprising determining a metric indicating a likelihood of success to localize the user using the map.

17. The method of claim 16 , wherein the metric is determined based on a co-visibility of a point of interest that is associated with different camera positions.

18. The method of claim 16 , wherein the metric is for one of a plurality of cells, and each of the cells represents a three dimensional space of a portion of the environment, wherein the camera system is configured to obtain multiple images, the multiple images comprising the image for which the score is determined, and wherein the metric is determined for one of the plurality of cells by: identifying a subset of the images that belong to a same range of viewing directions; determining respective scores for the images in the subset of the images; and summing the scores to obtain a total score.

19. A method performed by an apparatus that is configured to be worn on a head of a user, the apparatus having a screen configured to present graphics to the user, a camera system configured to view an environment in which the user is located, and a processing unit, the method comprising: obtaining, by the processing unit, output(s) from the camera system; determining a map by the processing unit based at least in part on the output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment; and obtaining, by the processing unit, a score for an image obtained from the camera system, the score indicating how well the image is constrained with respect to a map segment for forming the map, wherein the act of obtaining the score comprises computing the score or receiving the score, and wherein the image is well-constrained when the image forms a well-connected part of the map.

20. An apparatus configured to be worn on a head of a user, comprises: a screen configured to present graphics to the user; a camera system configured to view an environment in which the user is located; and a processing unit configured to determine a map based at least in part on output(s) from the camera system, wherein the map is configured for use by the processing unit to localize the user with respect to the environment, wherein the processing unit is configured to determine a score for an image obtained from the camera system, the score indicating how well the image is constrained with respect to a map segment for forming the map, and wherein the image is well-constrained when the image forms a well-connected part of the map.

Assignees

• Magic Leap Inc

Inventors

• Sharma Divya
• Shahrokni Ali
• Mohan Anush
• SINGHAL Prateek
• Zhao Xuan
• Sima Sergiu
• Langmann Benjamin

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06T15/005

  General purpose rendering architectures · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F1/1686

  the I/O peripheral being an integrated camera · CPC title