Video and lidar target detection and tracking system and method for segmenting moving targets

Patent metadata

Field	Value
Publication number	US-9111444-B2
Application number	US-201213665560-A
Country	US
Kind code	B2
Filing date	Oct 31, 2012
Priority date	Oct 31, 2012
Publication date	Aug 18, 2015
Grant date	Aug 18, 2015

Abstract

Embodiments of video and LIDAR target detection and tracking system and method for segmenting targets are generally described herein. In some embodiments, the system may perform moving target detection using three-frame difference processing on frames of LIDAR range images of a scene to identify a moving target and determine an initial position estimate for the moving target. The system may also perform segmentation of the moving target using a single frame of the LIDAR range images to generate a segmented target that may include a target mask and target centroid position. After segmentation, moving target registration including motion compensation may be performed by registering the segmented target in multiple LIDAR frames to determine a target position, a target orientation and a target heading. In some embodiments, a three-dimensional model of the moving target may be generated and added to a sequence of video frames.

First claim

What is claimed is:

1. A target detection and tracking system comprising:
a moving target detection module to perform moving target detection using three-frame difference processing on frames of LIDAR range images of a scene to identify a moving target and determine an initial position estimate for the moving target,
each LIDAR range image being formed from illumination of the scene with infrared laser light and analysis of round trip travel time of the infrared laser light collected from the scene,
each LIDAR range image comprising a depth map of the scene, each LIDAR range image being unaffected by shadows, surface markings, color, and lighting; and

a segmentation module to perform segmentation of the moving target using a single frame of the LIDAR range images to generate a segmented target including a target mask and target centroid position.

2. The system of claim 1 wherein the segmentation module is configured to:
transform the single frame of the LIDAR range images within a track window to a height image wherein pixel values of the height image correspond to heights of objects in the scene;
threshold out pixels of the height image to remove ground pixels and occlusions in the height image to generate a segmented target image;
apply a median filter to the single frame of the LIDAR range image to fill in gaps;
identify depth discontinuities in the LIDAR range image after the gaps are filled;
null-out pixels of the segmented target image that correspond to the depth discontinuities; and
establish boundaries within the segmented target image to identify potential targets.

3. The system of claim 2 wherein the segmentation module is further configured to:
filter the potential targets based on a predetermined size threshold to identify at least one target of interest; and
generate the segmented target image and a target mask for each of the identified targets of interest based on the filtering.

4. The system of claim 3 wherein the segmentation module is further configured to:
determine a three-dimensional (3D) target centroid for each of the targets of interest within the range image;
set the 3D target centroid as the target location; and
update a 3D velocity of the target.

5. The system of claim 4 wherein the segmentation module is further configured to determine an image gradient of the LIDAR range image after the gaps are filled in to identify the depth discontinuities, and
wherein the boundaries are established within the segmented target image after nulling-out the pixels that correspond to depth discontinuities to identify potential targets.

6. The system of claim 2 wherein the segmentation module is further configured to:
generate histogram values of the height image;
estimate a ground height from the histogram values; and
determine one or more height thresholds for the height image for use in thresholding out pixel values.

7. The system of claim 4 further comprising a motion compensation module,
wherein after segmentation, the motion compensation module is configured to perform moving target registration including motion compensation by registering the segmented target in multiple LIDAR frames to determine a target position, a target orientation and a target heading.

8. The system of claim 7 wherein the motion compensation module is further configured to perform an interactive closest point (ICP) to refine frame-to-frame registration.

9. A video and LIDAR target detection, tracking and segmentation method comprising:
performing moving target detection using three-frame difference processing on frames of LIDAR range images of a scene to identify a moving target and determine an initial position estimate for the moving target,
each LIDAR range image being formed from illumination of the scene with infrared laser light and analysis of round trip travel time of the infrared laser light collected from the scene,
each LIDAR range image comprising a depth map of the scene, each LIDAR range image being unaffected by shadows, surface markings, color, and lighting; and

performing segmentation of the moving target using a single frame of the LIDAR range images to generate a segmented target including a target mask and target centroid position.

10. The method of claim 9 wherein performing segmentation comprises:
transforming the single frame of the LIDAR range images within a track window to a height image wherein pixel values of the height image correspond to heights of objects in the scene;
thresholding-out pixels of the height image to remove ground pixels and occlusions in the height image to generate a segmented target image;
applying a median filter to the single frame of the LIDAR range image to fill in gaps;
identifying depth discontinuities in the LIDAR range image after the gaps are filled, the depth discontinuities being identified using an image gradient of the LIDAR range image;
nulling-out pixels of the segmented target image that correspond to the depth discontinuities; and
establishing boundaries within the segmented target image to identify potential targets.

11. The method of claim 10 wherein performing segmentation further comprises:
filtering the potential targets based on a predetermined size threshold to identify at least one target of interest; and
generating the segmented target image and a target mask for each of the identified targets of interest based on the filtering.

12. The method of claim 11 wherein performing segmentation further comprises:
determining a three-dimensional (3D) target centroid for each of the targets of interest within the range image;
setting the 3D target centroid as the target location; and
updating a 3D velocity of the target.

13. The method of claim 12 wherein performing segmentation further comprises determining an image gradient of the LIDAR range image after the gaps are filled in to identify the depth discontinuities, and
wherein the boundaries are established within the segmented target image after nulling-out the pixels that correspond to depth discontinuities to identify potential targets.

14. The method of claim 10 wherein the segmentation further comprises:
generating histogram values of the height image;
estimating a ground height from the histogram values; and
determining one or more height thresholds for the height image for use in thresholding out pixel values.

15. The method of claim 12 further comprising, after segmentation, performing moving target registration including motion compensation by registering the segmented target in multiple LIDAR frames to determine a target position, a target orientation and a target heading.

16. The method of claim 15 wherein performing moving target registration further comprises performing an interactive closest point (ICP) to refine frame-to-frame registration.

17. A method for target image segmentation comprising:
transforming a LIDAR range image within a track window to a height image wherein pixel values of the height image correspond to heights of objects in a scene,
the LIDAR range image being formed from illumination of the scene with infrared laser light and analysis of round trip travel time of the infrared laser light collected from the scene,
the LIDAR range image comprising a depth map of the scene,
the LIDAR range image being unaffected by shadows, surface markings, color, and lighting;

thresholding-out pixels of the height image to remove ground pixels and occlusions in the height image to generate a segmented target image;
applying a median filter to the LIDAR range image to fi

Assignees

• Raytheon Co

Inventors

• Kaganovich Vitaliy M

Classifications

• G06V10/255

  Detecting or recognising potential candidate objects based on visual cues, e.g. shapes · CPC title

• G06T2207/20032

  Median filtering · CPC title

• G01S7/4808

  Evaluating distance, position or velocity data · CPC title

• G06T2207/20036

  Morphological image processing · CPC title

• G06T7/2053

  Physics · mapped topic