Data fusion system for a vehicle equipped with unsynchronized perception sensors

Patent metadata

Field	Value
Publication number	US-11435752-B2
Application number	US-201815935796-A
Country	US
Kind code	B2
Filing date	Mar 26, 2018
Priority date	Mar 23, 2018
Publication date	Sep 6, 2022
Grant date	Sep 6, 2022

Abstract

A sensor data fusion system for a vehicle with multiple sensors includes a first-sensor, a second-sensor, and a controller-circuit. The first-sensor is configured to output a first-frame of data and a subsequent-frame of data indicative of objects present in a first-field-of-view. The first-frame is characterized by a first-time-stamp, the subsequent-frame of data characterized by a subsequent-time-stamp different from the first-time-stamp. The second-sensor is configured to output a second-frame of data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view. The second-frame is characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp. The controller-circuit is configured to synthesize an interpolated-frame from the first-frame and the subsequent-frame. The interpolated-frame is characterized by an interpolated-time-stamp that corresponds to the second-time-stamp. The controller-circuit fuses the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp, and operates the host-vehicle in accordance with the fused-frame.

First claim

We claim:

1. A sensor data fusion system for a host-vehicle with multiple sensors, said system comprising:
a camera mounted on the host-vehicle, said camera configured to output a first-frame of camera data and a subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;
a LiDAR mounted on the host-vehicle, said LiDAR configured to output a second-frame of point-cloud data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of point-cloud data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and
a controller-circuit in communication with the camera and the LiDAR, said controller-circuit configured to:
synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the point-cloud data of the second-frame;
fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;
determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and
operate the host-vehicle in accordance with the three-dimensional model of the environment.

2. The system of claim 1 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.

3. The system of claim 1 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.

4. The system of claim 1 , wherein the controller-circuit is further configured to:
synthesize the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.

5. A controller-circuit for a sensor data fusion system for a host-vehicle with multiple sensors, said controller-circuit comprising:
a first-input configured to communicate with a camera mounted on the host-vehicle, said camera configured to output a first-frame of camera data and a subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;
a second-input configured to communicate with a LiDAR mounted on the host-vehicle, said LiDAR configured to output a second-frame of point-cloud data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of point-cloud data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp; and
a processor in communication with the camera and the LiDAR, said processor configured to:
synthesize an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that the both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the point-cloud of the second-frame;
fuse the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp,
determine a three-dimensional model of an environment surrounding the host-vehicle based on the fused-frame; and
operate the host-vehicle in accordance with the three-dimensional model of the environment.

6. The controller-circuit of claim 5 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.

7. The controller-circuit of claim 5 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.

8. The controller-circuit of claim 5 , wherein the processor is further configured to:
synthesize the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.

9. A method of operating a sensor data fusion system for a host-vehicle with multiple sensors, said method comprising:
receiving a first-frame of camera data and a subsequent-frame of camera data from a camera mounted on the host-vehicle, said first-frame of camera data and said subsequent-frame of camera data indicative of objects present in a first-field-of-view, said first-frame of camera data characterized by a first-time-stamp, said subsequent-frame of camera data characterized by a subsequent-time-stamp different from the first-time-stamp;
receiving a second-frame of LiDAR data from a LiDAR mounted on the host-vehicle, said second-frame of LiDAR data indicative of objects present in a second-field-of-view that overlaps the first-field-of-view, said second-frame of LiDAR data characterized by a second-time-stamp temporally located between the first-time-stamp and the subsequent-time-stamp;
synthesizing an interpolated-frame from the first-frame and the subsequent-frame using motion flow analysis responsive to determining that both the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by greater than a time-threshold, said interpolated-frame characterized by an interpolated-time-stamp that corresponds to the second-time-stamp, such that the interpolated-frame is temporally synchronized with the LiDAR data of the second-frame;
fusing the interpolated-frame with the second-frame to provide a fused-frame of data characterized by the interpolated-time-stamp;
determining a three-dimensional model of an environment based on the fused-frame; and
operating the host-vehicle in accordance with the three-dimensional model of the environment.

10. The method of claim 9 , wherein the objects present in the first-field-of-view are in a travel path of the host-vehicle.

11. The method of claim 9 , wherein the objects present in the second-field-of-view are in a travel path of the host-vehicle.

12. The method of claim 9 , further comprising:
synthesizing the interpolated-frame from the first-frame responsive to determining that the first-time-stamp and the subsequent-time-stamp differ from the second-time-stamp by less than five milliseconds, wherein the interpolated-frame is the same as the first-frame.

13. A sensor data fusion system for a host-vehicle with multiple sensors, said system comprising:
a LiDAR mounted on the host-vehicle, said LiDAR configured to output a first-frame of point-cloud data and a subsequent-frame of point cloud data indicative of objects present in a first-field-of-view, said first-frame of point cloud data characterized by a first-time-stamp, said subsequent-frame of point-cloud data characterized by a subsequent-time-stamp different from the first-time-stamp;
a camera mounted on the host-vehicle, said cameraconfigured to output a secon

Assignees

• Motional Ad Llc

Inventors

• Ozbilgin Guchan
• Xu Wenda
• Snider Jarrod M
• Wang Yimu
• Yang Yifan
• Wei Junqing

Classifications

• G01S17/86 Primary

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06F18/251

  of input or preprocessed data · CPC title

• G01S17/88

  Lidar systems specially adapted for specific applications · CPC title