Gravity estimation and bundle adjustment for visual-inertial odometry

Patent metadata

Field	Value
Publication number	US-11328475-B2
Application number	US-202017072825-A
Country	US
Kind code	B2
Filing date	Oct 16, 2020
Priority date	Oct 18, 2019
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

Examples of the disclosure describe systems and methods for presenting virtual content on a wearable head device. In some embodiments, a state of a wearable head device is determined by minimizing a total error based on a reduced weight associated with a reprojection error. A view reflecting the determined state of the wearable head device is presented via a display of the wearable head device. In some embodiments, a wearable head device calculates a first preintegration term and second preintegration term based on the image data received via a sensor of the wearable head device and the inertial data received via a first IMU and a second IMU of the wearable head device. The wearable head device estimates a position of the device based on the first and second preintegration terms, and the wearable head device presents the virtual content based on the position of the device.

First claim

What is claimed is:

1. A method comprising:
receiving, via a sensor of a wearable head device, first sensor data indicative of a first feature in a first position;
receiving, via the sensor, second sensor data indicative of the first feature in a second position;
receiving, via an inertial measurement unit (IMU) on the wearable head device, inertial measurements;
determining a velocity based on the inertial measurements;
estimating a third position of the first feature based on the first position and the velocity;
determining a reprojection error based on the third position and the second position;
reducing a weight associated with the reprojection error;
determining a state of the wearable head device, wherein determining the state comprises minimizing a total error, and wherein the total error is based on the reduced weight associated with the reprojection error; and
presenting, via a display of the wearable head device, a view reflecting the determined state of the wearable head device.

2. The method of claim 1 , wherein the weight associated with the reprojection error is reduced based on a quantity of identified features.

3. The method of claim 1 , wherein reducing the weight associated with the reprojection error comprises determining if the reprojection error comprises a statistical outlier.

4. The method of claim 1 , wherein the total error is further based on an error from the inertial measurement unit.

5. The method of claim 1 , further comprising:
determining if the wearable head device is static for at least a threshold amount of time;
in accordance with a determination that the wearable head device is not static for at least the threshold amount of time, updating an information matrix; and
in accordance with a determination that the wearable head device is static for at least the threshold amount of time, forgoing updating the information matrix.

6. The method of claim 1 , further comprising pre-integrating the inertial measurements.

7. The method of claim 6 , further comprising:
receiving, via a second IMU on the wearable head device, second inertial measurements; and
pre-integrating the second inertial measurements.

8. The method of claim 6 , wherein:
the first sensor data is received at a first time, and
the inertial measurements being pre-integrated are associated with a movement of the wearable head device between the first time and a second time, the method further comprising:
determining whether a time interval between the first time and the second time is greater than a maximum time interval;
in accordance with a determination that the time interval is not greater than the maximum time interval, using the pre-integrated inertial measurements; and
in accordance with a determination that the time interval is greater than the maximum time interval, forgoing using the pre-integrated inertial measurements.

9. The method of claim 1 , further comprising applying a correction to the state based on an updated state of the wearable head device.

10. A system comprising:
a sensor of a wearable head device;
an inertial measurement unit of the wearable head device;
a display of the wearable head device;
one or more processors configured to execute a method comprising:
receiving, via the sensor of the wearable head device, first sensor data indicative of a first feature in a first position;
receiving, via the sensor, second sensor data indicative of the first feature in a second position;
receiving, via the inertial measurement unit on the wearable head device, inertial measurements;
determining a velocity based on the inertial measurements;
estimating a third position of the first feature based on the first position and the velocity;
determining a reprojection error based on the third position and the second position;
reducing a weight associated with the reprojection error;
determining a state of the wearable head device, wherein determining the state comprises minimizing a total error, and wherein the total error is based on the reduced weight associated with the reprojection error; and
presenting, via the display of the wearable head device, a view reflecting the determined state of the wearable head device.

11. The system of claim 10 , wherein the weight associated with the reprojection error is reduced based on a quantity of identified features.

12. The system of claim 10 , wherein reducing the weight associated with the reprojection error comprises determining if the reprojection error comprises a statistical outlier.

13. The system of claim 10 , wherein the total error is further based on an error from the inertial measurement unit.

14. The system of claim 10 , wherein the method further comprises:
determining if the wearable head device is static for at least a threshold amount of time;
in accordance with a determination that the wearable head device is not static for at least the threshold amount of time, updating an information matrix; and
in accordance with a determination that the wearable head device is static for at least the threshold amount of time, forgoing updating the information matrix.

15. The system of claim 10 , wherein the method further comprises pre-integrating the inertial measurements.

16. The system of claim 15 , wherein the method further comprises:
receiving, via a second IMU on the wearable head device, second inertial measurements; and
pre-integrating the second inertial measurements.

17. The system of claim 15 , wherein:
the first sensor data is received at a first time, and
the inertial measurements being pre-integrated are associated with a movement of the wearable head device between the first time and a second time, and
the method further comprises:
determining whether a time interval between the first time and the second time is greater than a maximum time interval;
in accordance with a determination that the time interval is not greater than the maximum time interval, using the pre-integrated inertial measurements; and
in accordance with a determination that the time interval is greater than the maximum time interval, forgoing using the pre-integrated inertial measurements.

18. The system of claim 10 , wherein the method further comprises applying a correction to the state based on an updated state of the wearable head device.

19. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to execute a method comprising:
receiving, via a sensor of a wearable head device, first sensor data indicative of a first feature in a first position;
receiving, via the sensor, second sensor data indicative of the first feature in a second position;
receiving, via an inertial measurement unit on the wearable head device, inertial measurements;
determining a velocity based on the inertial measurements;
estimating a third position of the first feature based on the first position and the velocity;
determining a reprojection error based on the third position and the second position;
reducing a weight associated with the reprojection error;
determining a state of the wearable head device, wherein determining the state comprises minimizing a total error, and wherein the total error is based on the reduced weight associated with the reprojection error; and
presenting, via a display of the wearable head device, a view reflecting the determined state of the wearable head device.

20. The non-transitory computer-readable medium

Assignees

• Magic Leap Inc

Inventors

• Huang Yu-Hsiang
• Levine Evan Gregory
• Napolskikh Igor
• Kasper Dominik Michael
• Sanchez Nicuesa Manel Quim
• Sima Sergiu
• Langmann Benjamin
• Swaminathan Ashwin
• Zahnert Martin Georg
• Czuprynski Blazej Marek
• Faro Joao Antonio Pereira
• Tobler Christoph
• Ghasemalizadeh Omid

Classifications

• G01P15/08

  with conversion into electric or magnetic values · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G01P15/00

  Measuring acceleration; Measuring deceleration; Measuring shock, i.e. sudden change of acceleration · CPC title

• G06T15/20 Primary

  Perspective computation · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title