Localization and mapping utilizing visual odometry

Patent metadata

Field	Value
Publication number	US-11527011-B2
Application number	US-202117486664-A
Country	US
Kind code	B2
Filing date	Sep 27, 2021
Priority date	Aug 9, 2019
Publication date	Dec 13, 2022
Grant date	Dec 13, 2022

Abstract

In one embodiment, a method includes determining correspondence data between a sequence of images based on identified features in the sequence of images and predicted pose based on motion data, and determining current state information based on the correspondence data and the motion data. The current state information comprises at least a current pose of the wearable device relative to the environment capture by the one or more cameras. Furthermore, the method comprises receiving map points in a three-dimensional map and their associated descriptors based on the identified features in the sequence of images and identifying one or more of the map points in the sequence of images based on the associated descriptors associated with the map points. The current state information is further determined based on the identified one or more of the map points.

First claim

What is claimed is:

1. A method comprising, by a computing system:
receiving, at an IMU integration unit, motion data captured by one or more motion sensors of a wearable device;
generating, at the IMU integration unit, a predicted pose of the wearable device based on the motion data of the wearable device;
receiving, at a tracking unit, a sequence of images of an environment captured by one or more cameras;
identifying, at the tracking unit, features in the sequence of images;
determining, at the tracking unit, correspondence data between the sequence of images based on the identified features in the sequence of images and the predicted pose received from the IMU integration unit;
determining, at a filter unit, current state information of the wearable device based on the correspondence data received from the tracking unit and the motion data received from the IMU integration unit, the current state information comprising at least a current pose of the wearable device relative to the environment captured by the one or more cameras;
retrieving, by a mapping unit of the computing system, regional map data from a remote map server using (1) the features identified by the tracking unit in the sequence of images of the environment and (2) the current state information of the wearable device determined at the filter unit, the regional map data being associated with a portion of a three-dimensional map hosted by the remote server;
receiving, at the tracking unit and from the mapping unit, map points and associated descriptors for the map points from the regional map data retrieved from the remote map server; and
identifying, at the tracking unit, one or more of the map points in the sequence of images based on one or more of the associated descriptors associated with the one or more of the received map points, wherein the determining of the current state information is further based on the identified one or more of the map points within the sequence of images;
wherein the mapping unit of the computing system performs the retrieval of the regional map data from the remote server at a second frequency that is relatively lower than a first frequency of the tracking unit at which the tracking unit identifies the features in the sequence of images of the environment and determines correspondences between the sequences of images based on the identified features.

2. The method of claim 1 , wherein the determining of the correspondence data comprises:
identifying a first feature in a first image of the sequence of images; and
searching, in a second image of the sequence of images, for a second feature that corresponds to the first feature in the first image;
wherein the searching is performed along an epipolar line segment determined using the predicted pose.

3. The method of claim 1 , wherein the current state information is determined based on an aggregation of the motion data.

4. The method of claim 1 , wherein the current state information is determined using an optimization algorithm.

5. The method of claim 1 , wherein the current state information is used to generate a next predicted pose of the wearable device.

6. The method of claim 1 , wherein the IMU integration unit operates at a higher frequency than the tracking unit and the filter unit.

7. The method of claim 1 , wherein the wearable device is an augmented-reality device, wherein the method further comprises:
rendering augmented-reality content based on the current pose.

8. The method of claim 1 , wherein the current state information further comprises a current position of the wearable device relative to the three-dimensional map.

9. The method of claim 1 ,
wherein the mapping unit is configured to operate on demand or at a lower frequency than the IMU integration unit, the tracking unit, and the filter unit.

10. The method of claim 9 ,
wherein the IMU integration unit is located within a head-mounted device; and
wherein the tracking unit, the filter unit, and the mapping unit are implemented in a local computing device separated from the head-mounted device.

11. The method of claim 10 , wherein the local computing device comprises one or more processors, wherein the one or more processors are configured to implement the tracking unit, the filter unit, and the mapping unit.

12. The method of claim 9 ,
wherein the IMU integration unit, the tracking unit, and the filter unit are located within a head-mounted device; and
wherein the mapping unit is implemented in a local computing device separated from the head-mounted device.

13. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive motion data captured by one or more motion sensors of a wearable device;
generate a predicted pose of the wearable device based on the motion data of the wearable device;
receive a sequence of images of an environment captured by one or more cameras;
identify features in the sequence of images;
determine correspondence data between the sequence of images based on the identified features in the sequence of images and the predicted pose;
determine current state information of the wearable device based on the correspondence data and the motion data, the current state information comprising at least a current pose of the wearable device relative to the environment captured by the one or more cameras;
retrieve regional map data from a remote map server using (1) the features identified by the tracking unit in the sequence of images of the environment and (2) the current state information of the wearable device, the regional map data being associated with a portion of a three-dimensional map hosted by the remote server;
receive map points and associated descriptors for the map points from the regional map data retrieved from the remote map server; and
identify one or more of the map points in the sequence of images based on one or more of the associated descriptors associated with the one or more of the received map points, wherein the determining of the current state information is further based on the identified one or more of the map points within the sequence of images;
wherein a mapping unit performs the retrieval of the regional map data from the remote server at a second frequency that is relatively lower than a first frequency of a tracking unit at which the tracking unit identifies the features in the sequence of images of the environment and determines correspondences between the sequences of images based on the identified features.

14. The media of claim 13 , wherein to determine the correspondence data, the software is operable when executed to:
identify a first feature in a first image of the sequence of images; and
search, in a second image of the sequence of images, for a second feature that corresponds to the first feature in the first image;
wherein the searching is performed along an epipolar line segment determined using the predicted pose.

15. The media of claim 13 , wherein the current state information is determined based on an aggregation of the motion data.

16. The media of claim 13 , wherein the current state information is determined using an optimization algorithm.

17. The media of claim 13 , wherein the current state information is used to generate a next predicted pose of the wearable device.

18. A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one

Assignees

• Meta Platforms Tech Llc

Inventors

• Engel Jakob Julian
• Mourikis Anastasios
• Mur Artal Raul

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G01P13/00

  Indicating or recording presence, absence, or direction, of movement (electric switches H01H; counting moving objects G06M7/00) · CPC title

• G06T7/74 Primary

  involving reference images or patches · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title