Visual odometry in autonomous machine applications

Patent metadata

Field	Value
Publication number	US-11803192-B2
Application number	US-202217900622-A
Country	US
Kind code	B2
Filing date	Aug 31, 2022
Priority date	Dec 1, 2019
Publication date	Oct 31, 2023
Grant date	Oct 31, 2023

Abstract

Systems and methods for performing visual odometry more rapidly. Pairs of representations from sensor data (such as images from one or more cameras) are selected, and features common to both representations of the pair are identified. Portions of bundle adjustment matrices that correspond to the pair are updated using the common features. These updates are maintained in register memory until all portions of the matrices that correspond to the pair are updated. By selecting only common features of one particular pair of representations, updated matrix values may be kept in registers. Accordingly, matrix updates for each common feature may be collectively saved with a single write of the registers to other memory. In this manner, fewer write operations are performed from register memory to other memory, thus reducing the time required to update bundle adjustment matrices and thus speeding the bundle adjustment process.

First claim

What is claimed is:

1. A method comprising:
identifying one or more features common to at least two representations of a plurality of representations generated using sensor data, the sensor data being generated using one or more sensors corresponding to an autonomous or semi-autonomous machine;
determining one or more values of a bundle adjustment for at least a portion of the sensor data generated using at least one sensor of the one or more sensors, the one or more values corresponding to the one or more features identified as common to the at least two representations;
determining at least one visual odometry parameter corresponding to one or more sensors using the one or more values of the bundle adjustment; and
using the at least one visual odometry parameter to determine at least one of a position or an orientation corresponding to the autonomous or semi-autonomous machine.

2. The method of claim 1 , wherein the determining the one or more values of the bundle adjustment includes determining the one or more values for one or more portions of bundle adjustment matrices that correspond to the at least two representations.

3. The method of claim 1 , wherein the one or more values of the bundle adjustment are stored using one or more registers, and the one or more values of the bundle adjustment are transferred from the one or more registers to one or more memories using a single write operation, wherein the at least one visual odometry parameter is determined using the one or more values of the bundle adjustment transferred to the one or more memories.

4. The method of claim 1 , wherein the determining the one or more values of the bundle adjustment includes updating at least one of:
one or more feature values corresponding to the one or more features,
or one or more camera parameters corresponding to the one or more features using a cost function.

5. The method of claim 1 , wherein at least one feature value of the one or more features identified as common to the at least two representations are stored using sequential addresses of one or more memories, wherein the determining the one or more values of the bundle adjustment includes accessing the one or more feature values stored using the one or more memories.

6. The method of claim 5 , wherein the one or more features include a plurality of features, processor threads are sequentially allocated to the plurality of features, and the processor threads allocated to the plurality of features perform the determining the one or more values of the bundle adjustment.

7. The method of claim 1 , wherein the at least two representations comprise images generated using a camera and the features comprise image points corresponding to the images.

8. The method of claim 1 , wherein the identifying the one or more features common to the at least two representations includes assembling lists of features for the at least two representations and comparing at least two lists of the lists of features to determine common elements between the at least two lists of features.

9. The method of claim 1 , wherein the at least one visual odometry parameter includes at least one of:
one or more camera poses,
or one or more feature positions.

10. The method of claim 1 , wherein the determining the one or more values of the bundle adjustment uses one or more first memories comprising one or more register memories, the one or more values of the bundle adjustment are written from the one or more first memories to one or more second memories, and the determining the at least one visual odometry parameter is based at least on accessing the one or more values of the bundle adjustment stored in the one or more second memories.

11. A processor comprising:
one or more circuits to:
determine at least one visual odometry parameter corresponding to at least one sensor using one or more values of a bundle adjustment based at least on:
identifying one or more features common to two or more images generated using sensor data, and determining the one or more values of the bundle adjustment corresponding to the one or more features identified as common to the two or more images, and

use the at least one visual odometry parameter to determine at least one of a position or an orientation corresponding to an autonomous or semi-autonomous machine.

12. The processor of claim 11 , wherein the determining the one or more values of the bundle adjustment includes determining the one or more values for one or more portions of bundle adjustment matrices that correspond to the two or more images.

13. The processor of claim 11 , wherein the one or more values of the bundle adjustment are stored using one or more registers, and the one or more values of the bundle adjustment are transferred from the one or more registers to one or more memories using a single write operation, wherein the at least one visual odometry parameter is determined using the one or more values of the bundle adjustment transferred to the one or more memories.

14. The processor of claim 11 , wherein the determining the one or more values of the bundle adjustment includes updating at least one of:
one or more feature values corresponding to the one or more features,
or one or more camera parameters corresponding to the one or more features using a cost function.

15. The processor of claim 11 , one or more feature values of the one or more features identified as common to the two or more images are stored using sequential addresses of one or more memories, and the determining the one or more values of the bundle adjustment includes accessing the one or more feature values stored using the one or more memories.

16. A system comprising:
parallel processing circuitry to perform operations including:
identifying one or more features common to two or more images generated using sensor data, the sensor data being generated using one or more sensors corresponding to an autonomous or semi-autonomous machine;
determining one or more values of a bundle adjustment corresponding to the one or more features identified as common to the two or more images;
determining at least one visual odometry parameter corresponding to one or more sensors using the one or more values of the bundle adjustment; and
using the at least one visual odometry parameter to determine at least one of a position or an orientation corresponding to the autonomous or semi-autonomous machine.

17. The system of claim 16 , wherein the determining the values of the bundle adjustment includes determining the one or more values for one or more portions of bundle adjustment matrices that correspond to the two or more images.

18. The system of claim 16 , wherein the one or more values of the bundle adjustment are stored in one or more registers, the one or more values of the bundle adjustment are transferred from the one or more registers to one or more memories using a single write operation, and the at least one visual odometry parameter is determined using the one or more values of the bundle adjustment transferred to the one or more memories.

19. The system of claim 16 , wherein the determining the one or more values of the bundle adjustment includes updating at least one of one or more feature values corresponding to the one or more features or one or more camera parameters corresponding to the one or more features using a cost function.

20. The system of claim 16 , wherein one or more feature values of the one or more features identified as common to the two or more images are

Assignees

• Nvidia Corp

Inventors

• Grabner Michael
• Furtek Jeremy
• Nister David

Classifications

• G05D1/0253 Primary

  extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70) · CPC title

• G05D1/0251

  extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593) · CPC title

• G06T7/246 Primary

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/285

  using a sequence of stereo image pairs · CPC title

• G06T7/35

  using statistical methods · CPC title