Calibration of sensor systems

Patent metadata

Field	Value
Publication number	US-11360197-B2
Application number	US-202016869506-A
Country	US
Kind code	B2
Filing date	May 7, 2020
Priority date	Jan 7, 2020
Publication date	Jun 14, 2022
Grant date	Jun 14, 2022

Abstract

A system includes multiple sensors having different, and overlapping, fields of regard and a controller communicatively coupled to sensors. Methods for calibrating the multiple sensors include obtaining data from the sensors, determining optimized transformation parameters for at least one of the sensors, and transforming the data from one sensor projection plane to the other sensor projection plane. The method is an iterative process that uses the metric of mutual information between the data sets, and transformed data sets of the sensors, to determine an optimized set of transformation parameters. The sensors may be a plurality of lidar sensors, a camera and a lidar sensor, or other sets of sensors.

First claim

What is claimed is:

1. A method for processing data received from sensors, the method comprising:
receiving, by processing hardware from a first sensor, first sensor data corresponding to an environment detected in a field of regard (FOR) of the first sensor;
receiving, by the processing hardware from a second sensor having a physical offset in position and/or orientation from the first sensor, second sensor data corresponding to the environment detected in a FOR of the second sensor at least partially overlapping the FOR of the first sensor; and
determining, by the processing hardware, a set of transformation parameters for alignment of the first sensor data and the second sensor data, including iteratively modifying candidate transformation parameters in view of a metric of mutual information between the first sensor data and the second sensor data, wherein determining the metric of mutual information includes generating a joint histogram between the first sensor data and the second sensor data for a region of overlap between the FOR of the first sensor and the FOR of the second sensor.

2. The method of claim 1 , wherein:
receiving the first sensor data includes receiving a two-dimensional image made up of pixels each including a respective intensity value; and
receiving the second sensor data includes receiving a three-dimensional point cloud made up of points each including a set of spatial coordinates and a respective intensity value.

3. The method of claim 2 , wherein the metric of mutual information indicates correlation between the intensity values in the first sensor data and the intensity values in the second sensor data.

4. The method of claim 1 , wherein:
receiving the first sensor data includes receiving a first three-dimensional (3D) point cloud made up of points each including a set of spatial coordinates; and
receiving the second sensor data includes receiving a second 3D point cloud made up of points each including a set of spatial coordinates; and
wherein the metric of mutual information indicates correlation between the set of spatial coordinates in the first sensor data and the set of spatial coordinates in the second sensor data.

5. The method of claim 4 , wherein:
each of the points in the first 3D point cloud and in the second 3D point cloud further includes a respective intensity or reflectivity value; and
the metric of mutual information further indicates correlation between the intensity values in the first sensor data and the intensity values in the second sensor data.

6. The method of claim 5 , further comprising applying a first weight to the sets of spatial coordinates and a second weight to the intensity values when generating the metric of mutual information.

7. The method of claim 1 , wherein the set of transformation parameters include at least one of rotation or translation offsets.

8. The method of claim 1 , wherein iteratively modifying the candidate transformation parameters includes:
setting, by the processing hardware, a candidate set of transformation parameters to an initial set of the transformation parameters;
transforming, by the processing hardware, the second sensor data into a reference frame of the first sensor data, in accordance with the candidate set of the transformation parameters;
calculating, by the processing hardware, the metric of mutual information for the first sensor data and the transformed second sensor data; and
repeating the transforming and the calculating, for different respective candidate sets of transformation parameters, until an exit condition is satisfied.

9. The method of claim 8 , further comprising:
receiving the initial set of transformation parameters from an operator via a user interface.

10. The method of claim 8 , further comprising:
selecting a result of a previous calibration process as the initial set of transformation parameters.

11. The method of claim 8 , further comprising:
selecting the initial set of transformation parameters using a grid search that yields an overlap between the FOR of the first sensor and the FOR of the second sensor above a certain threshold value.

12. The method of claim 8 , wherein transforming the second sensor data includes applying a homogeneous transformation matrix, wherein the homogeneous transformation matrix is based on a rotation quaternion and/or a translation vector.

13. The method of claim 8 , wherein calculating the metric of mutual information includes:
determining a region of overlap between the FOR of the first sensor and the FOR of the second sensor, and
determining the metric of mutual information only for the region of overlap.

14. The method of claim 8 , further comprising
repeating the transforming and the calculating includes maximizing the metric of mutual information based on the joint histogram.

15. The method of claim 8 , wherein repeating the transforming and the calculating includes:
applying, by the processing hardware, a gradient-descent algorithm to the candidate set of transformation parameters to generate an updated candidate set of transformation parameters.

16. The method of claim 15 , wherein the gradient-descent algorithm conforms to a Barzilai-Borwein method.

17. The method of claim 8 , wherein repeating the transforming and the calculating includes:
applying, by the processing hardware, a gradient-free algorithm to the candidate set of transformation parameters to generate an updated candidate set of transformation parameters.

18. The method of claim 17 , wherein the gradient-free function conforms to a Nelder-Mead optimization method.

19. The method of claim 8 , wherein the exit condition corresponds to at least one of:
(i) the metric of mutual information exceeding a first predefined threshold value,
(ii) a cost value based on the metric of mutual information being below a second predefined threshold value,
(iii) a number of iterations of the transforming and the calculating exceeding a third predefined threshold value,
(iv) a difference between a cost of a current iteration and a cost of a previous iteration being below a fourth predefined threshold value.

20. The method of claim 1 , wherein the joint histogram includes a two-dimensional grid of values, wherein each axis of the grid represents intensity or depth data from one of the sensors, and each value of the grid corresponds to a probability or a number of times that a corresponding sample pair of data from the two sensors is observed in the region of overlap.

21. The method of claim 1 , wherein determining the metric of mutual information further includes:
applying a smoothing filter to the joint histogram; and
calculating the metric of mutual information by applying a probabilistic technique to values of the joint histogram.

22. The method of claim 1 , wherein each of the first sensor data and the second sensor data includes multiple frames captured by the respective sensor.

23. The method of claim 1 , wherein:
receiving the first sensor data includes receiving a first 3D point cloud; and
receiving the second sensor data includes receiving a second 3D point cloud;
the method further comprising:
generating a plurality of projections of the first 3D point cloud and the second 3D point cloud onto respective projection planes;
generating a respective metric of mutual information for each of the projection planes, wherein the metric of mutual information for a projection plane indicates correlation be

Assignees

• Luminar Llc

Inventors

• Sutavani Amey
• Walajapet Mohan Lekha
• Englard Benjamin

Classifications

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/32

  using correlation-based methods · CPC title

• G01S17/931

  of land vehicles · CPC title

• G01S17/89

  for mapping or imaging · CPC title