Sensor calibration validation

Patent metadata

Field	Value
Publication number	US-12323574-B2
Application number	US-202217739530-A
Country	US
Kind code	B2
Filing date	May 9, 2022
Priority date	May 9, 2022
Publication date	Jun 3, 2025
Grant date	Jun 3, 2025

Abstract

Techniques for determining a probability that a first sensor is miscalibrated with respect a second sensor are discussed herein. For example, a computing device may receive calibrated extrinsics of a camera to a lidar, determine a plurality of sets of perturbed extrinsics based on the calibrated extrinsics, determine respective costs for perturbed extrinsics of the plurality of sets of perturbed extrinsics based on image data captured by the camera, the plurality of sets of perturbed extrinsics, and lidar data captured by the lidar, and determine a local maxima score for the calibrated extrinsics based at least in part on the respective costs for the perturbed extrinsics of the plurality of sets of perturbed extrinsics and a cost of the calibrated extrinsics. The computing device may then determine a probability that the camera is miscalibrated based on a Bayes probability and the local maxima score.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more non-transitory computer readable media storing computer executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving current extrinsics of a camera relative to a lidar associated with an autonomous vehicle, the current extrinsics comprising a current extrinsic parameter associated with one or more of a roll, pitch, or yaw between the camera and the lidar;
determining a set of perturbed extrinsics, a perturbed extrinsic of the set of perturbed extrinsics having a perturbed extrinsic parameter which differs from the current extrinsic parameter by a multiple of a step size and less than or equal to a maximum difference;
determining a current cost based on an image from the camera, lidar data from the lidar, and the current extrinsics;
determining a set of costs associated with the set of perturbed extrinsics based on the image, the lidar data, and the set of perturbed extrinsics;
determining, based at least in part on the current cost and the set of costs, a local maxima score for the current extrinsics;
receiving, as a distribution of calibrated extrinsics, a kernel density estimation distribution of a set of calibrated extrinsics, a calibrated extrinsic of the set of calibrated extrinsics having a local maxima score less than a local maxima score threshold;
determining a probability that the camera is miscalibrated based on the local maxima score for the current extrinsics and the distribution of calibrated extrinsics; and
based at least in part on determining that the probability that the camera is miscalibrated is below a threshold, operating the autonomous vehicle based at least in part on the current extrinsics.

2. The system of claim 1 , wherein the local maxima score for the current extrinsics is associated with a percentage of the set of costs that are less than the current cost associated with the current extrinsics.

3. The system of claim 1 , wherein determining the probability that the camera is miscalibrated is further based on a distribution of known miscalibrated extrinsics that is based on additional ground truth extrinsics determined for having respective local maxima scores greater than or equal to the local maxima score threshold.

4. The system of claim 1 , wherein determining the probability is further based at least in part on a Bayesian probability and the distribution.

5. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving current extrinsics representing a calibration between a first sensor and a second sensor, the first sensor and the second sensor associated with a vehicle;
determining a set of perturbed extrinsics based on the current extrinsics;
determining a set of costs associated with the set of perturbed extrinsics based on first data captured by the first sensor, the set of perturbed extrinsics, and second data captured by the second sensor;
determining a local maxima score for the current extrinsics based at least in part on the set of costs and a cost associated with the current extrinsics;
determining a probability that the first sensor is miscalibrated based on the local maxima score and based on a previously computed first distribution of first extrinsics associated with good calibrations having respective local maxima scores less than a local maxima score threshold; and
operating, based at least in part on determining that the probability that the first sensor is miscalibrated is less than or equal to a threshold, the vehicle based at least in part on the current extrinsics.

6. The one or more non-transitory computer-readable media of claim 5 , wherein determining the set of perturbed extrinsics comprises varying a parameter of the current extrinsics by a range of values.

7. The one or more non-transitory computer-readable media of claim 6 , wherein the range of values differ by a fixed step size and wherein the previously computed first distribution of first extrinsics is a kernel density distribution.

8. The one or more non-transitory computer-readable media of claim 6 , wherein the parameter comprises one or more of a roll, a pitch, or a yaw between the first sensor and second sensor.

9. The one or more non-transitory computer-readable media of claim 5 , wherein the local maxima score is associated with a percentage of the set of costs that are less than the cost of the current extrinsics, and
wherein determining the probability comprises determining a Bayes probability.

10. The one or more non-transitory computer-readable media of claim 5 , wherein determining the probability that the first sensor is miscalibrated is further based on a second distribution of second extrinsics associated with known miscalibrated extrinsics.

11. The one or more non-transitory computer-readable media of claim 5 , wherein the first sensor is a camera and the second sensor is a lidar.

12. A method comprising:
receiving current extrinsics representing a calibration between a first sensor and a second sensor, the first sensor and the second sensor associated with a vehicle;
determining a set of perturbed extrinsics based on the current extrinsics;
determining a set of costs associated with the set of perturbed extrinsics based on data captured by the first sensor and the set of perturbed extrinsics;
determining a local maxima score for the current extrinsics based at least in part on the set of costs and a cost associated with the current extrinsics;
determining a probability that the first sensor is miscalibrated based on the local maxima score and based on a previously computed first distribution of first extrinsics associated with good calibrations; and
operating, based at least in part on determining that the probability that the first sensor is miscalibrated is less than or equal to a threshold, the vehicle based at least in part on the current extrinsics.

13. The method of claim 12 , wherein determining the set of perturbed extrinsics comprises varying a parameter of the current extrinsics by a range of values.

14. The method of claim 13 , wherein the range of values differ by a fixed step size.

15. The method of claim 13 , wherein the parameter comprises one or more of a roll, a pitch, or a yaw between the first sensor and the second sensor, and
wherein the previously computed first distribution of extrinsics is associated with local maxima scores less than or equal to a threshold maxima score.

16. The method of claim 12 , wherein the local maxima score is associated with a percentage of the set of costs that are less than the cost of the current extrinsics, and
wherein determining the probability comprises determining a Bayes probability.

17. The method of claim 12 , wherein determining the probability that the first sensor is miscalibrated is further based on a second distribution of second extrinsics associated with known miscalibrated extrinsics.

18. The system of claim 1 , wherein determining the set of perturbed extrinsics comprises varying a parameter of the current extrinsics by a range of values.

19. The one or more non-transitory computer-readable media of claim 5 , wherein operating the vehicle comprises determining a trajectory for the vehicle to traverse an environment based at least in part on the current extrinsics.

20. The system of claim 1 , wherein the camera

Assignees

• Zoox Inc

Inventors

• Adams Derek
• Hashemifar Zakieh Sadat
• Mesolongitis Agis Iakovos

Classifications

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10024

  Color image · CPC title

• B60W2050/0215

  Sensor drifts or sensor failures · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title