Visual odometry in autonomous machine applications

Patent metadata

Field	Value
Publication number	US-11435756-B2
Application number	US-202017108965-A
Country	US
Kind code	B2
Filing date	Dec 1, 2020
Priority date	Dec 1, 2019
Publication date	Sep 6, 2022
Grant date	Sep 6, 2022

Abstract

Systems and methods for performing visual odometry more rapidly. Pairs of representations from sensor data (such as images from one or more cameras) are selected, and features common to both representations of the pair are identified. Portions of bundle adjustment matrices that correspond to the pair are updated using the common features. These updates are maintained in register memory until all portions of the matrices that correspond to the pair are updated. By selecting only common features of one particular pair of representations, updated matrix values may be kept in registers. Accordingly, matrix updates for each common feature may be collectively saved with a single write of the registers to other memory. In this manner, fewer write operations are performed from register memory to other memory, thus reducing the time required to update bundle adjustment matrices and thus speeding the bundle adjustment process.

First claim

What is claimed is:

1. A method of determining sensor parameters, the method comprising:
selecting a pair of representations generated from sensor data;
for the selected pair, identifying features common to each representation of the selected pair using parallel processing circuitry;
for each of the features, determining corresponding values of a bundle adjustment process, and storing the determined values in a first memory that is a register memory;
writing the determined values to a second memory responsive to determining the corresponding values for at least one of the features; and
determining at least one sensor parameter by performing the bundle adjustment process using the determined values.

2. The method of claim 1 , further comprising storing the identified features at sequential addresses of a memory.

3. The method of claim 2 , further comprising sequentially allocating a processor thread for each of the stored features.

4. The method of claim 1 , wherein the determining corresponding values of the bundle adjustment process further comprises determining corresponding values of a portion of a Hessian matrix, the portion corresponding to the selected pair of representations.

5. The method of claim 4 , wherein the method further comprises allocating a processing element of a graphics processing unit (GPU) for determining the corresponding values of the portion of the Hessian matrix.

6. The method of claim 1 , wherein the representations comprise images generated using a camera, wherein the features comprise image points.

7. The method of claim 1 , wherein the identifying features common to each representation of the selected pair further comprises, for each representation of the selected pair of representations, assembling a list of features of each representation, and comparing the lists of the selected pair of representations to determine common elements of the lists.

8. The method of claim 1 , wherein the sensor parameters correspond to camera parameters, and wherein at least one sensor parameter comprises at least one of: one or more camera poses and one or more feature positions.

9. The method of claim 1 , wherein the determining at least one parameter of the sensor further comprises determining positions of the features by performing the bundle adjustment process using the determined values.

10. A method of determining camera poses and feature positions, the method comprising:
performing a bundle adjustment process using parallel processing circuitry, by:
identifying features common to both images of a pair of images;
using a first memory that is a register memory, determining one or more values of a matrix of the bundle adjustment process corresponding to the identified common features;
after the determining, writing the determined values from the first memory to a second memory; and
determining at least one of the poses of the camera using the determined values.

11. The method of claim 10 , further comprising storing the identified features at sequential addresses of a memory.

12. The method of claim 11 , further comprising sequentially allocating a processor thread for each of the stored features.

13. The method of claim 10 , wherein the matrix is a Hessian matrix.

14. The method of claim 10 , wherein the method further comprises allocating a processing element of a graphics processing unit (GPU) for performing the determining one or more values of the matrix of the bundle adjustment process corresponding to the identified common features.

15. The method of claim 10 , wherein the features comprise image points.

16. The method of claim 10 , wherein the identifying further comprises, for each image of the pair of images, assembling a list of features of the each image, and comparing the lists of the selected pair of images to determine common elements of the lists.

17. The method of claim 10 , wherein the images are images of a camera of a vehicle.

18. The method of claim 10 , wherein the determining at least the poses of the camera further comprises determining positions of the features by performing the bundle adjustment process using the determined values.

19. A system for determining sensor parameters, the system comprising:
parallel processing circuitry configured to:
select a pair of representations generated from sensor data;
for the selected pair, identify features common to each representation of the selected pair using the parallel processing circuitry;
for each of the features, determine corresponding values of a bundle adjustment process, and store the determined values in a first memory that is a register memory;
write the determined values to a second memory responsive to determining the corresponding values for at least one of the features; and
determine at least one sensor parameter by performing the bundle adjustment process using the determined values.

20. The system of claim 19 , wherein the parallel processing circuitry is further configured to store the identified features at sequential addresses of a memory.

21. The system of claim 20 , wherein the parallel processing circuitry is further configured to sequentially allocate a processor thread for each of the stored features.

22. The system of claim 19 , wherein the determining corresponding values of the bundle adjustment process further comprises determining corresponding values of a portion of a Hessian matrix, the portion corresponding to the selected pair of representations.

23. The system of claim 22 , wherein the parallel processing circuitry is further configured to allocate a processing element of a graphics processing unit (GPU) for determining the corresponding values of the portion of the Hessian matrix.

24. The system of claim 19 , wherein the representations comprise images generated using a camera, wherein the features comprise image points.

25. The system of claim 19 , wherein the identifying features common to each representation of the selected pair further comprises, for each representation of the selected pair of representations, assembling a list of features of each representation, and comparing the lists of the selected pair of representations to determine common elements of the lists.

26. The system of claim 19 , wherein the sensor parameters correspond to camera parameters, and wherein at least one sensor parameter comprises at least one of: one or more camera poses and one or more feature positions.

27. The system of claim 19 , wherein the determining at least one parameter of the sensor further comprises determining positions of the features by performing the bundle adjustment process using the determined values.

Assignees

• Nvidia Corp

Inventors

• Grabner Michael
• Furtek Jeremy
• Nister David

Classifications

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G05D1/0253 Primary

  extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/35

  using statistical methods · CPC title