System and method for concurrent odometry and mapping

Patent metadata

Field	Value
Publication number	US-11734846-B2
Application number	US-202016875488-A
Country	US
Kind code	B2
Filing date	May 15, 2020
Priority date	May 18, 2016
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

An electronic device tracks its motion in an environment while building a three-dimensional visual representation of the environment that is used to correct drift in the tracked motion. A motion tracking module estimates poses of the electronic device based on feature descriptors corresponding to the visual appearance of spatial features of objects in the environment. A mapping module builds a three-dimensional visual representation of the environment based on a stored plurality of maps, and feature descriptors and estimated device poses received from the motion tracking module. The mapping module provides the three-dimensional visual representation of the environment to a localization module, which identifies correspondences between stored and observed feature descriptors. The localization module performs a loop closure by minimizing the discrepancies between matching feature descriptors to compute a localized pose. The localized pose corrects drift in the estimated pose generated by the motion tracking module.

First claim

What is claimed is:

1. A method comprising:
tracking, at a head mounted device, motion of the head mounted device relative to an environment of the head mounted device based on one or more spatial features representing one or more objects in the environment;
estimating, at the head mounted device, a pose of the head mounted device based on the one or more spatial features representing the one or more objects in the environment;
generating, at the head mounted device, a three-dimensional visual representation of the environment of the head mounted device based on the one or more spatial features;
periodically updating the generated three-dimensional visual representation of the environment based on one or more feature descriptors of the one or more spatial features representing one or more objects in the environment; and
correcting, at the head mounted device, one or more drift errors in the estimated pose of the head mounted device using the updated generated three-dimensional visual representation of the environment.

2. The method of claim 1 , further comprising:
capturing, at the head mounted device, image sensor data using one or more visual sensors integrated with the head mounted device; and
capturing, at the head mounted device, non-image sensor data using one or more non-image sensors integrated with the head mounted device.

3. The method of claim 2 , further comprising generating the one or more spatial features based on the captured image sensor data from the one or more visual sensors and the captured non-image sensor data from the one or more non-image sensors integrated with the head mounted device.

4. The method of claim 2 , further comprising generating, at the head mounted device, the one or more feature descriptors of the one or more spatial features representing one or more objects in the environment based on the captured image sensor data from the one or more visual sensors and the captured non-image sensor data from the one or more non-image sensors integrated with the head mounted device.

5. The method of claim 4 , wherein correcting, at the head mounted device, the one or more drift errors comprises performing, at the head mounted device, a loop-closure between the one or more generated feature descriptors and the updated generated three-dimensional visual representation of the environment.

6. The method of claim 4 , wherein correcting, at the head mounted device, the one or more drift errors is based in part on comparing the one or more generated feature descriptors to one or more stored feature descriptors.

7. The method of claim 1 , further comprising generating, at the head mounted device, a localized pose of the head mounted device based on correcting the one or more drift errors in the estimated pose of the head mounted device.

8. A head mounted device, comprising:
a motion tracking module configured to track motion of the head mounted device relative to an environment of the head mounted device based on one or more spatial features representing one or more objects in the environment, wherein the motion tracking module is further to estimate a pose of the head mounted device based on the one or more spatial features representing the one or more objects in the environment;
a mapping module configured to generate and periodically update a three-dimensional visual representation of the environment of the head mounted device based on the one or more spatial features and one or more feature descriptors of the one or more spatial features; and
a loop closure module configured to correct one or more drift errors in the estimated pose of the head mounted device using the updated generated three-dimensional visual representation of the environment.

9. The head mounted device of claim 8 , further comprising:
a visual sensor configured to capture image sensor data; and
a non-image sensor configured to capture non-image sensor data.

10. The head mounted device of claim 9 , wherein the motion tracking module is further to generate the one or more spatial features based on the captured image sensor data from the visual sensors and the captured non-image sensor data from the non-image sensor.

11. The head mounted device of claim 9 , wherein the motion tracking module is further to generate the one or more feature descriptors of the one or more spatial features representing one or more objects in the environment based on the captured image sensor data from the visual sensor and the captured non-image sensor data from the non-image sensor.

12. The head mounted device of claim 11 , wherein the loop closure module is further to perform a loop-closure between the one or more generated feature descriptors and the updated generated three-dimensional visual representation of the environment.

Assignees

• Google Llc

Inventors

• Nerurkar Esha
• Lynen Simon
• Zhao Sheng

Classifications

• G06T7/55 Primary

  from multiple images · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/038

  Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry · CPC title

• G06T7/73

  using feature-based methods · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title