Visual inertial odometry localization using sparse sensors

Patent metadata

Field	Value
Publication number	US-12249092-B2
Application number	US-202117493620-A
Country	US
Kind code	B2
Filing date	Oct 4, 2021
Priority date	Sep 15, 2020
Publication date	Mar 11, 2025
Grant date	Mar 11, 2025

Abstract

In one embodiment, a computing system accesses a set of 3D locations associated with features in an environment previously captured by a camera from a previous camera pose. The computing system determines a predicted camera pose using the previous camera pose and motion measurements generated using a motion sensor associated with the camera. The computing system projects the set of 3D locations toward the predicted camera pose and onto a 2D image plane associated with the camera. The computing system generates, based on the projected set of 3D locations on the 2D image plane, an activation map specifying a subset of the pixel sensors of the camera that are to be activated. The computing system instructs, using the activation map, the camera to activate the subset of pixel sensors to capture a new image of the environment. The computing system reads pixel values of the new image.

First claim

What is claimed is:

1. A method comprising, by a computing system:
accessing a set of 3D locations associated with features in an environment previously captured by a camera from a previous camera pose;
determining a predicted camera pose using the previous camera pose and motion measurements generated using a motion sensor associated with the camera;
projecting the set of 3D locations toward the predicted camera pose and onto a 2D image plane associated with the camera;
generating, based on the projected set of 3D locations on the 2D image plane, an activation map specifying a subset of the pixel sensors of the camera that are to be activated;
instructing, using the activation map, the camera to activate the subset of pixel sensors to capture a new image of the environment;
reading pixel values of the new image corresponding to the subset of pixel sensors activated by the camera; and
tracking the features in the environment based on the pixel values.

2. The method of claim 1 , further comprising:
receiving, from the camera, a plurality of frames containing a plurality of images of the environment; and
processing a set of the plurality of images to identify the features in the environment, wherein the features are indicative of a landmark in the environment.

3. The method of claim 1 , further comprising:
accessing one or more intrinsic camera parameters of the camera and one or more camera lens distortion characteristics, wherein projecting the set of 3D locations toward the predicted camera pose and onto the 2D image plane is based on the one or more intrinsic camera parameters of the camera and the one or more camera lens distortion characteristics.

4. The method of claim 1 , further comprising:
generating an occupancy grid based on the projection of the set of 3D locations toward the predicted camera pose and onto the 2D image plane, wherein the occupancy grid indicates one or more cells within the occupancy grid that are occupied by the projection of the set of 3D locations toward the predicted camera pose and onto the 2D image plane, and wherein generating the activation map is based on the occupancy grid.

5. The method of claim 4 , further comprising:
analyzing one or more unoccupied cells in the occupancy grid; and
determining whether to include the one or more unoccupied cells in the activation map based on a predetermined framerate.

6. The method of claim 5 , wherein the camera captures a plurality of images at a first framerate greater than the predetermined framerate.

7. The method of claim 1 , further comprising:
generating one or more search lines based on the projection of the set of 3D locations toward the predicted camera pose and onto the 2D image plane, wherein the one or more search lines are included in the activation map.

8. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access a set of 3D locations associated with features in an environment previously captured by a camera from a previous camera pose;
determine a predicted camera pose using the previous camera pose and motion measurements generated using a motion sensor associated with the camera;
project the set of 3D locations toward the predicted camera pose and onto a 2D image plane associated with the camera;
generate, based on the projected set of 3D locations on the 2D image plane, an activation map specifying a subset of the pixel sensors of the camera that are to be activated;
instruct, using the activation map, the camera to activate the subset of pixel sensors to capture a new image of the environment;
read pixel values of the new image corresponding to the subset of pixel sensors activated by the camera; and
track the features in the environment based on the pixel values.

9. The media of claim 8 , wherein the one or more computer-readable non-transitory storage media is further operable when executed to:
receive, from the camera, a plurality of frames containing a plurality of images of the environment; and
process a set of the plurality of images to identify the features in the environment, wherein the features are indicative of a landmark in the environment.

10. The media of claim 8 , wherein the one or more computer-readable non-transitory storage media is further operable when executed to:
access one or more intrinsic camera parameters of the camera and one or more camera lens distortion characteristics, wherein projecting the set of 3D locations toward the predicted camera pose and onto the 2D image plane is based on the one or more intrinsic camera parameters of the camera and the one or more camera lens distortion characteristics.

11. The media of claim 8 , wherein the one or more computer-readable non-transitory storage media is further operable when executed to:
generate an occupancy grid based on the projection of the set of 3D locations toward the predicted camera pose and onto the 2D image plane, wherein the occupancy grid indicates one or more cells within the occupancy grid that are occupied by the projection of the set of 3D locations toward the predicted camera pose and onto the 2D image plane, and wherein generating the activation map is based on the occupancy grid.

12. The media of claim 11 , wherein the one or more computer-readable non-transitory storage media is further operable when executed to:
analyze one or more unoccupied cells in the occupancy grid; and
determine whether to include the one or more unoccupied cells in the activation map based on a predetermined framerate.

13. The media of claim 12 , wherein the camera captures a plurality of images at a first framerate greater than the predetermined framerate.

14. The media of claim 8 , wherein the one or more computer-readable non-transitory storage media is further operable when executed to:
generate one or more search lines based on the projection of the set of 3D locations toward the predicted camera pose and onto the 2D image plane, wherein the one or more search lines are included in the activation map.

15. A system comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
access a set of 3D locations associated with features in an environment previously captured by a camera from a previous camera pose;
determine a predicted camera pose using the previous camera pose and motion measurements generated using a motion sensor associated with the camera;
project the set of 3D locations toward the predicted camera pose and onto a 2D image plane associated with the camera;
generate, based on the projected set of 3D locations on the 2D image plane, an activation map specifying a subset of the pixel sensors of the camera that are to be activated;
instruct, using the activation map, the camera to activate the subset of pixel sensors to capture a new image of the environment;
read pixel values of the new image corresponding to the subset of pixel sensors activated by the camera; and
track the features in the environment based on the pixel values.

16. The system of claim 15 , wherein the instructions are further executable by the one or more processors to:
receive, from the camera, a plurality of frames containing a plurality of images of the environment; and
process a set of the plurality of images to identify the features in the environment, wherein the features are indicative of a landmark in the environment.

1

Assignees

• Meta Platforms Tech Llc

Inventors

• Suleiman Amr
• Mourikis Anastasios
• Alaghi Armin
• Berkovich Andrew Samuel
• Alkalay Shlomo
• Kal Muzaffer
• Lee Vincent
• Newcombe Richard Andrew

Classifications

• G06T2200/04

  involving 3D image data · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• H04N25/44

  by partially reading an SSIS array · CPC title

• G06T2207/30244

  Camera pose · CPC title