Systems and methods for generic visual odometry using learned features via neural camera models

Patent metadata

Field	Value
Publication number	US-12333750-B2
Application number	US-202217722360-A
Country	US
Kind code	B2
Filing date	Apr 17, 2022
Priority date	Sep 15, 2020
Publication date	Jun 17, 2025
Grant date	Jun 17, 2025

Abstract

Systems and methods for self-supervised learning for visual odometry using camera images, may include: estimating correspondences between keypoints of a target camera image and keypoints of a context camera image; based on the keypoint correspondences, lifting a set of 2D keypoints to 3D, using a neural camera model; and projecting the 3D keypoints into the context camera image using the neural camera model. Some embodiments may use the neural camera model to achieve the lifting and projecting of keypoints without a known or calibrated camera model.

First claim

What is claimed is:

1. A method comprising:
estimating correspondences between keypoints of a target camera image and keypoints of a context camera image, wherein the target camera image and the context camera image are obtained from a monocular sequence;
using a ray surface decoder to predict a ray surface from the target image, wherein the predicted ray surface associates a respective pixel in the target image with a corresponding direction;
based on the keypoint correspondences and the predicted ray surface, lifting a set of 2D keypoints to 3D, using a neural camera model; and
projecting the 3D keypoints into the context camera image using the neural camera model.

2. The method of claim 1 , wherein lifting to 3D and projection to 2D achieved through the neural camera model are performed without a known or calibrated camera model.

3. The method of claim 1 , wherein the neural camera model is configured to learn a pixel-wise ray surface that enables learning depth and pose estimates in a self-supervised way from a wider variety of camera geometries.

4. The method of claim 1 , further comprising using a key point network to learn a keypoint matrix for the target camera image and the context camera image, wherein estimating the correspondences between the keypoints of the target camera image and the keypoints of the context camera image comprises using learned descriptors from the keypoint matrix to estimate the correspondences between the keypoints of the target camera image and the keypoints of the context camera image.

5. The method of claim 4 , wherein using the learned descriptors to estimate the correspondences between the target camera image and the context camera image comprises computing a set of corresponding keypoints comprising a keypoint from the target image and a warped corresponding keypoint in the context image.

6. The method of claim 4 , wherein using the learned descriptors to estimate the correspondences between the target image and the context image comprises self-supervised 3D keypoint learning.

7. The method of claim 1 , wherein using the ray surface decoder to predict the ray surface from the target image comprises learning a residual ray surface and adding the residual ray surface to a fixed ray surface to produce the predicted ray surface.

Assignees

• Toyota Res Inst Inc

Inventors

• Guizilini Vitor
• Vasiljevic Igor
• Ambrus Rares A
• Pillai Sudeep
• Gaidon Adrien

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T15/205

  Image-based rendering · CPC title

• G06T7/73

  using feature-based methods · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title