Approaches for determining sensor calibration

Patent metadata

Field	Value
Publication number	US-11238370-B2
Application number	US-201816237321-A
Country	US
Kind code	B2
Filing date	Dec 31, 2018
Priority date	Dec 31, 2018
Publication date	Feb 1, 2022
Grant date	Feb 1, 2022

Abstract

Systems, methods, and non-transitory computer-readable media can determine first sensor data captured by a first sensor of a vehicle. Second sensor data captured by a second sensor of the vehicle can be determined. Information describing the first sensor data and the second sensor data can be provided to a machine learning model trained to predict whether a pair of sensors are calibrated or mis-calibrated based on sensor data captured by the pair of sensors. A determination is made whether the first sensor and the second sensor are calibrated or mis-calibrated based on an output from the machine learning model.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining, by a computing system, first sensor data captured by a first sensor of a vehicle;
determining, by the computing system, second sensor data captured by a second sensor of the vehicle;
providing, by the computing system, information describing the first sensor data and the second sensor data to a machine learning model,
wherein the machine learning model is trained based on at least first training data that is associated with a first category of positive training examples and second training data that is associated with a second category of negative training examples, and the second training data corresponds to a transformation of the first training data based on one or more offsets applied to the first training data, and wherein the second training data corresponds to the one or more off sets producing inaccuracies in the first training data;
determining, by the computing system, whether the first sensor and the second sensor are calibrated or mis-calibrated based on an output from the machine learning model that includes applying a regression analysis to the first sensor data and the second sensor data; and
automatically calibrating the first sensor and the second sensor in real-time according to the output.

2. The computer-implemented method of claim 1 , wherein the regression analysis outputs an amount of mis-calibration between the first sensor and the second sensor, and wherein the first sensor and the second sensor each correspond to an optical camera, a Light Detection And Ranging (LiDAR) sensor, a radar sensor, an infrared camera, an inertial navigation system (INS), or an ultrasound sensor.

3. The computer-implemented method of claim 1 , wherein the first sensor data captured by the first sensor and the second sensor data captured by the second sensor are synchronized based on time.

4. The computer-implemented method of claim 1 , wherein the first sensor is a same type of sensor as the second sensor.

5. The computer-implemented method of claim 1 , wherein the first sensor is a different type of sensor than the second sensor.

6. The computer-implemented method of claim 1 , wherein the first training data represents sensor data captured by a pair of calibrated sensors.

7. The computer-implemented method of claim 1 , wherein the second training data is inaccurate relative to the first training data.

8. The computer-implemented method of claim 1 , further comprising:
determining, by the computing system, that the first sensor and the second sensor are mis-calibrated based on the output from the machine learning model; and
determining, by the computing system, an amount of mis-calibration between the first sensor and the second sensor.

9. The computer-implemented method of claim 1 , further comprising:
determining, by the computing system, that the first sensor and the second sensor are mis-calibrated based on the output from the machine learning model; and
generating, by the computing system, one or more alerts in response to the determination that the first sensor and the second sensor are mis-calibrated.

10. The computer-implemented method of claim 1 , further comprising:
determining, by the computing system, that the first sensor and the second sensor are mis-calibrated based on the output from the machine learning model; and
applying, by the computing system, one or more operations to correct the mis-calibration based at least in part on an amount of mis-calibration offset outputted by the machine learning model.

11. A system comprising: at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform:

determining first sensor data captured by a first sensor of a vehicle; determining second sensor data captured by a second sensor of the vehicle;
providing information describing the first sensor data and the second sensor data to a machine learning model to predict whether the first sensor and the second sensor are calibrated or mis-calibrated based on at least the first sensor data and the second sensor data, wherein the machine learning model is trained based on at least first training data that is associated with a first category of positive training examples and second training data that is associated with a second category of negative training examples, and the second training data corresponds to a transformation of the first training data based on one or more offsets applied to the first training data, wherein the second training data corresponds to the one or more offsets producing inaccuracies in the first training data;
determining whether the first sensor and the second sensor are calibrated or mis-calibrated based on an output from the machine learning model that includes applying a regression analysis to the first sensor data and the second sensor data; and
automatically calibrating the first sensor and the second sensor in real-time according to the output.

12. The system of claim 11 , wherein the regression analysis outputs an amount of mis-calibration between the first sensor and the second sensor, and
wherein the first sensor and the second sensor each correspond to an optical camera, a Light Detection And Ranging (LiDAR) sensor, a radar sensor, an infrared camera, an inertial navigation system (INS), or an ultrasound sensor.

13. The system of claim 11 , wherein the first sensor data captured by the first sensor and the second sensor data captured by the second sensor are synchronized based on time.

14. The system of claim 11 , wherein the first sensor is a same type of sensor as the second sensor or the first sensor is a different type of sensor than the second sensor.

15. The system of claim 11 , wherein the first training data represents sensor data captured by a pair of calibrated sensors, and the sensor data is ground truth data.

16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:
determining first sensor data captured by a first sensor of a vehicle; determining second sensor data captured by a second sensor of the vehicle;
providing information describing the first sensor data and the second sensor data to a machine learning model to predict whether the first sensor and the second sensor are calibrated or mis-calibrated based on at least the first sensor data and the second sensor data, wherein the machine learning model is trained based on at least first training data that is associated with a first category of positive training examples and second training data that is associated with a second category of negative training examples, and the second training data corresponds to a transformation of the first training data based on one or more offsets applied to the first training data, wherein the second training data corresponds to the one or more off sets producing inaccuracies in the first training data; and
determining whether the first sensor and the second sensor are calibrated or mis-calibrated based on an output from the machine learning model that includes applying a regression analysis to the first sensor data and the second sensor data; and
automatically calibrating the first sensor and the second sensor in real-time according to the output.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the regression analysis outputs an amount of mis-calibration between the first sensor and the second sensor, and
wh

Assignees

• Woven Planet North America Inc

Inventors

• Jain Ashesh
• Zhang Lei

Classifications

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G01S17/931

  of land vehicles · CPC title