SYSTEM AND METHOD FOR FUSING RADAR/CAMERA OBJECT DATA AND LiDAR SCAN POINTS

Patent metadata

Field	Value
Publication number	US-2016018524-A1
Application number	US-201514828402-A
Country	US
Kind code	A1
Filing date	Aug 17, 2015
Priority date	Mar 15, 2012
Publication date	Jan 21, 2016
Grant date	—

Abstract

A system and method for fusing the outputs from multiple LiDAR sensors on a vehicle that includes cueing the fusion process in response to an object being detected by a radar sensor and/or a vision system. The method includes providing object files for objects detected by the LiDAR sensors at a previous sample time, where the object files identify the position, orientation and velocity of the detected objects. The method projects object models in the object files from the previous sample time to provide predicted object models. The method also includes receiving a plurality of scan returns from objects detected in the field-of-view of the sensors at a current sample time and constructing a point cloud from the scan returns. The method then segments the scan points in the point cloud into predicted scan clusters, where each cluster identifies an object detected by the sensors.

First claim

What is claimed is:

1 . A system for fusing outputs from multiple LiDAR sensors with other object data, said system comprising:
a plurality of LiDAR sensors configured to detect objects in proximity to a host entity; a radar sensor or a vision system configured to detect objects in proximity to the host entity; and a processor in communication with the LiDAR sensors and the radar sensor or the vision system, said processor being configured with an algorithm including steps of; providing object files for objects detected by the LiDAR sensors at a previous sample time, said object files having object models that identify a position, orientation and velocity of the detected objects by the sensors; tracking the object models of the objects detected by the sensors; projecting tracked object models in the object files from the previous sample time to provide predicted object models; receiving a plurality of scan returns from objects detected in a field-of-view of the sensors at a current sample time; constructing a scan point cloud from the scan returns; segmenting the scan points in the scan point cloud into predicted scan clusters where each scan cluster initially identifies an object detected by the LiDAR sensors; matching the predicted scan clusters with the predicted object models; reading object data from objects detected by the radar sensor or the vision system; registering the object data to a coordinate frame designated for the scan points of the LiDAR sensors; matching the object data with the object models in the object files using the predicted object models; merging predicted object models that have been identified as separate scan clusters in the previous sample time but are now identified as a single scan cluster in the current sample time using the matched predicted scan clusters and the matched object data; and splitting predicted object models that have been identified as a single scan cluster in the previous sample time but are now identified as separate scan clusters in the current sample time using the matched predicted scan clusters and the matched object data.

2 . The system according to claim 1 wherein segmenting the scan points in the point cloud includes separating the clusters of scan points in the point cloud so that clusters identify a separate object that is being tracked.

3 . The system according to claim 1 wherein matching the predicted scan clusters with the predicted object models includes generating a bipartite graph that matches the scan clusters to the predicted object models.

4 . The system according to claim 3 wherein generating a bipartite graph includes positioning the scan points at vertices of the graph.

5 . The system according to claim 3 wherein merging predicted object models and splitting predicted object models includes converting the bipartite graph to an induced bipartite graph showing the object models that are being merged or split.

6 . The system according to claim 5 wherein converting the bipartite graph to an induced bipartite graph includes computing weights and a cardinality of edges for the induced bipartite graph, where each weight identifies a positional change of one of the scan points for the at least one sensor to a location of a scan point for another one of the sensors and the edges define matching between object model points and segmented scan points.

7 . The system according to claim 6 wherein converting the bipartite graph to an induced bipartite graph includes comparing the cardinality of the edges to a threshold and highlighting the edge if the cardinality is greater than the threshold, and wherein an edge is removed if it not highlighted.

8 . The system according to claim 7 wherein deleting object models that are no longer present in the predicted scan clusters includes deleting an object model that was connected to an edge that is removed.

9 . The system according to claim 1 wherein providing object model updates includes identifying the object data, a transformation parameter, the object models and the scan map clusters in a Bayesian network for the previous time frame to the current time frame.

10 . A system for fusing outputs from multiple LiDAR sensors with other object data, said system comprising:
a plurality of LiDAR sensors configured to detect objects in proximity to a host entity; a radar sensor or a vision system configured to detect objects in proximity to the host entity; and a processor in communication with the LiDAR sensors and the radar sensor or the vision system, said processor being configured with an algorithm including steps of; providing object files for objects detected by the LiDAR sensors at a previous sample time, said object files having object models that identify a position, orientation and velocity of the objects detected by the sensors; tracking the object models of the objects detected by the sensors; projecting tracked object models in the object files from the previous sample time to provide predicted object models; receiving a plurality of scan returns from objects detected in a field-of-view of the sensors at a current sample time; constructing a scan point cloud from the scan returns; segmenting the scan points in the scan point cloud into predicted scan clusters where each scan cluster initially identifies an object detected by the sensors; matching the predicted scan clusters with the predicted object models; reading object data from objects detected by the radar sensor or the vision system; registering the object data to a coordinate frame designated for the scan points of the LiDAR sensors; matching, using a computer including a processor and memory, the object data with the object models in the object files using the predicted object models; merging predicted object models that have been identified as separate scan clusters in the previous sample time but are now identified as a single scan cluster in the current sample time using the matched predicted scan clusters and the matched object data; and splitting predicted object models that have been identified as a single scan cluster in the previous sample time but are now identified as separate scan clusters in the current sample time using the matched predicted scan clusters and the matched object data.

11 . The system according to claim 10 wherein segmenting the scan points in the point cloud includes separating the clusters of scan points in the point cloud so that the clusters identify a separate object that is being tracked.

12 . The system according to claim 10 wherein matching the predicted scan clusters with the predicted object models includes generating a bipartite graph that matches the scan clusters to the predicted object models.

13 . The system according to claim 12 wherein generating a bipartite graph includes positioning the scan points at vertices of the graph.

14 . The system according to claim 12 wherein merging predicted object models and splitting predicted object models includes converting the bipartite graph to an induced bipartite graph showing the object models that are being merged and split.

15 . The system according to claim 14 wherein converting the bipartite graph to an induced bipartite graph includes computing weights and a cardinality of edges for the induced bipartite graph, where each weight identifies a positional change of one of the scan points for the at least one sensor to a location of a scan point for another one of the sensors and the edges define matching between object model points and segmented scan points.

16 . The syste

Assignees

• Gm Global Tech Operations Inc

Inventors

• Zeng Shuqing

Classifications

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S17/931

  of land vehicles · CPC title

• G01S17/66 Primary

  Tracking systems using electromagnetic waves other than radio waves · CPC title

• G01S17/87 Primary

  Combinations of systems using electromagnetic waves other than radio waves · CPC title

• G01S17/936

  Physics · mapped topic