System and method for fusing outputs of sensors having different resolutions

Patent metadata

Field	Value
Publication number	US-10582121-B2
Application number	US-201614993307-A
Country	US
Kind code	B2
Filing date	Jan 12, 2016
Priority date	Jan 12, 2016
Publication date	Mar 3, 2020
Grant date	Mar 3, 2020

Abstract

A method for fusing measurements of sensors having different resolutions performs jointly a calibration of the sensors and a fusion of the their measurements to produce calibration parameters defining a geometrical mapping between coordinate systems of the sensors and a fused set of measurements that includes the modality of a sensor with resolution greater than its resolution. The calibration and the fusion are performed jointly to update the calibration parameters and the fused set of measurements in dependence on each other.

First claim

We claim:

1. A method for fusing measurements of sensors having different resolutions, comprising:
acquiring a first set of measurements of a scene measured by a first sensor;
acquiring a second set measurements of the scene measured by a second sensor, wherein a modality of the first sensor differs from a modality of the second sensor, and wherein a resolution of the first sensor is less than a resolution of the second sensor; and
performing jointly a calibration of the first and the second sensors, and a fusion of the first and the second set of measurements to produce calibration parameters defining a geometrical mapping between coordinate systems of the first and the second sensors, and a fused set of measurements that includes an upsampled first set of measurement of the modality of the first sensor with resolution greater than the resolution of the first sensor,
wherein the calibration and the fusion are performed jointly and iteratively to update the calibration parameters and the upsampled first set of measurements in dependence on each other until a termination condition is met, wherein, during an iteration, the fusion upsamples the first set of measurements guided by values of the second set of measurements related to the first set of measurements according to the calibration parameters to produce the upsampled first set of measurements that minimizes a cost function that determines an alignment error between the upsampled first set of measurements and the second set of measurements, and the calibration updates the calibration parameters based on the cost function, such that, for the next iteration, the upsampled first set of measurements determined using the updated calibration parameters reduces the alignment error according to the cost function, wherein steps of the method are performed using a processor.

2. The method of claim 1 , wherein the joint calibration and fusion minimizes the cost function given the calibration parameters.

3. The method of claim 1 , wherein the first and the second sensors are installed at a vehicle for acquiring the first and the second sets of measurements.

4. The method of claim 1 , wherein the first sensor is a depth sensor and the second sensor is a camera, wherein the first set of measurements is a depth image with the resolution of the depth sensor, wherein the second set of measurements is an intensity image with the resolution of the camera, and wherein the fused set of measurements is a depth image with the resolution of the camera.

5. A system, comprising:
a first sensor to measure a scene to produce a first set of measurements of the scene;
a second sensor to measure the scene to produce a second set measurements of the scene, wherein a resolution of the first sensor is less than a resolution of the second sensor; and
a processor to perform jointly, (1) a calibration of the first and the second sensors, and (2) a fusion of the first and the second set of measurements to produce calibration parameters defining a geometrical mapping between coordinate systems of the first and the second sensors and a fused set of measurements that includes an upsampled first set of measurement of a modality of the first sensor with resolution greater than the resolution of the first sensor, wherein the calibration and the fusion are performed jointly to update the calibration parameters and the fused set of measurements in dependence on each other, wherein the fusion upsamples the first set of measurements to produce the upsampled first set of measurements using values of the second set measurements, such that the joint calibration and fusion optimizes a cost function given the calibration parameters, wherein the calibration and the fusion are performed jointly and iteratively to update the calibration parameters and the upsampled first set of measurements in dependence on each other until a termination condition is met, wherein during a current iteration the fusion upsamples the first set of measurements guided by values of the second set of measurements, related to the first set of measurements according to the calibration parameters, to produce the upsampled first set of measurements that minimizes a cost function that determines an alignment error between the upsampled first set of measurements and the second set of measurements, and the calibration updates the calibration parameters based on the cost function, such that, for the next iteration, the upsampled first set of measurements determined using the updated calibration parameters reduces the alignment error according to the cost function.

6. The system of claim 5 , wherein the processor is configured to track an object in the scene using the fused set of measurements.

7. The system of claim 5 , wherein the first and the second sensors are installed at a vehicle for acquiring the first and the second sets of measurements, and wherein the processor forms a part of a computational system of the vehicle.

8. The system of claim 5 , wherein the first sensor is a depth sensor and the second sensor is a camera, wherein the first set of measurements is a depth image with the resolution of the depth sensor, wherein the second set of measurements is an intensity image with the resolution of the camera, and wherein the fused set of measurements is a depth image with the resolution of the camera.

9. A method for fusing outputs of uncalibrated sensors, comprising:
acquiring a first set of measurements of a scene from a first sensor;
acquiring a second set measurements of the scene from a second sensor, wherein a resolution of the first sensor is less than a resolution of the second sensor;
performing jointly, (1) a calibration of the first and the second sensors, and (2) a fusion of the first and the second set of measurements, wherein the calibration and the fusion are performed jointly and iteratively to update the calibration parameters and the fused set of measurements in dependence on each other until a termination condition is met, wherein during a current iteration the fusion upsamples the first set of measurements guided by values of the second set of measurements, related to the first set of measurements according to the calibration parameters, to produce an upsampled first set of measurements that minimizes a cost function that determines an alignment error between the upsampled first set of measurements and the second set of measurements, and the calibration updates the calibration parameters based on the cost function, such that, for the next iteration, the upsampled first set of measurements determined using the updated calibration parameters reduces the alignment error according to the cost function.

10. The method of claim 9 , wherein the modality of the first sensor differs from a modality of the second sensor, wherein the fused set of measurements includes data having the modality of the first sensor and the modality of the second sensor, and has a resolution of the second sensor.

11. The method of claim 10 , wherein the first sensor is a depth sensor and the second sensor is a camera.

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Kamilov Ulugbek
• Castorena Juan
• Boufounos Petros

Classifications

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S7/4808 Primary

  Evaluating distance, position or velocity data · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title