Fusion of RGB images and LiDAR data for lane classification

Patent metadata

Field	Value
Publication number	US-9710714-B2
Application number	US-201514816808-A
Country	US
Kind code	B2
Filing date	Aug 3, 2015
Priority date	Aug 3, 2015
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

Point cloud data is received and a ground plane is segmented. A two-dimensional image of the segmented ground plane is generated based on intensity values of the segmented ground plane. Lane marking candidates are determined based on intensity within the generated two-dimensional image. Image data is received and the generated two-dimensional image is registered with the received image data. Lane marking candidates of the received image data are determined based on the lane marking candidates of the registered two-dimensional image. Image patches are selected from the two-dimensional image and from the received image data based on the determined lane markings. Feature maps including selected image patches from the registered two-dimensional image and received data are generated. The set of feature maps are sub-sampled, and a feature vector is generated based on the set of feature maps. Lane markings are determined from the generated feature vector.

First claim

We claim:

1. A method comprising:
receiving point cloud data;
segmenting a ground plane from the point cloud data;
generating a two-dimensional image of the segmented ground plane based on intensity values of the segmented ground plane;
determining lane marking candidates based on intensity within the generated two-dimensional image;
receiving image data;
registering the generated two-dimensional image with the received image data;
determining lane marking candidates of the received image data based on the determined lane marking candidates of the registered two-dimensional image; and
selecting a plurality of image patches from the registered two-dimensional image and from the registered image data based on the determined lane markings.

2. The method of claim 1 , further comprising:
generating a plurality of feature maps from the plurality of image patches wherein the plurality of image patches includes selected image patches from the registered two-dimensional image and selected image patches from the received image data;
sub-sampling the plurality of feature maps;
generating a feature vector based on the plurality of feature maps; and
determining lane markings from the generated feature vector based on intensity.

3. The method of claim 1 , wherein determining lane marking candidates within the further comprises:
applying a positive threshold to determine lane marking candidates; and
applying a negative threshold to determine negative lane marking candidates, wherein the positive threshold is higher than the negative threshold.

4. The method of claim 1 , wherein registering the generated two-dimensional image with the received image data is based on lane markings.

5. The method of claim 1 , further comprising:
increasing the contrast of the generated two-dimensional image based on intensity values of the segmented point cloud data.

6. The method of claim 1 , wherein segmenting the ground plane further comprises one of thresholding based on height, the normal of a plurality of points of the three-dimensional point cloud, or plane fitting.

7. The method of claim 1 , further comprising
smoothing gaps between projected points on the generated two-dimensional image.

8. The method of claim 1 , wherein determining lane marking candidates based on intensity within the generated two-dimensional image comprises:
computing the convex hull of white pixels of the generated two-dimensional image.

9. The method of claim 2 , wherein generating the plurality of feature maps further comprises:
determining the classification of each pixel based on a sliding window.

10. The method of claim 1 , wherein the received image data comprises at least one color image.

11. The method of claim 10 , further comprising;
transforming a color space of the received image data.

12. The method of claim 1 , wherein the received point cloud data and the received image data include geoposition information and pose information.

13. The method of claim 1 , wherein the point cloud data and the image data are collected simultaneously.

14. An apparatus comprising:
at least one processor; and
at least one memory including computer program code for one or more programs; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:
receive point cloud data;
segment a ground plane from the point cloud data;
generate a two-dimensional image of the segmented ground plane based on intensity values of the segmented ground plane;
determine lane marking candidates based on intensity within the two-dimensional image;
receive image data;
register the generated two-dimensional image with the received image data;
determine lane marking candidates of the received image data based on the determined lane marking candidates of the registered two-dimensional image;
select a plurality of image patches from the two-dimensional image and from the received image data based on the determined lane markings;
generate a plurality of feature maps from the plurality of image patches wherein the plurality of image patches includes selected image patches from the registered two-dimensional image and selected image patches from the received image data;
sub-sample the set of feature maps;
generate a feature vector based on the set of feature maps; and
determine lane markings from the generated feature vector based on intensity.

15. The apparatus of claim of claim 14 , the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:
apply a positive threshold to determine the presence of lane marking candidates; and
apply a negative threshold to determine an absence of lane marking candidates, wherein the positive threshold is higher than the negative threshold.

16. The apparatus of claim 14 , the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:
increase the contrast of the generated two-dimensional image based on intensity values of the segmented point cloud data.

17. The apparatus of claim 14 , the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform:
smooth gaps between projected points on the generated two-dimensional image.

18. A non-transitory computer readable medium including instructions that when executed are operable to:
receive a plurality of color images;
receive point cloud data;
generate a plurality of two-dimensional images based on intensity of the point cloud data;
register each of the generated two-dimensional images with color images of the plurality of color images based on geolocation and pose;
generate a plurality of feature maps from each registered two-dimensional image and its corresponding color image based on a classifier; wherein the classifier is based on positive lane marking examples and negative lane marking examples;
sub-sample the plurality of feature maps;
generate a feature vector based on the plurality of feature maps; and
determine lane markings from the generated feature vector based on intensity.

19. The non-transitory computer readable medium of claim 18 , including instructions that when executed are operable to:
smooth gaps between projected points on the determined lane markings.

20. The non-transitory computer readable medium of claim 18 , including instructions that when executed are operable to:
apply the classifier to each pixel of the of each registered two-dimensional image and its corresponding color image based on a sliding window.

Assignees

• Nokia Technologies Oy

Inventors

• Chen Xin
• Zang Andi
• Huang Xinyu

Classifications

• G06V10/806

  of extracted features · CPC title

• G06V20/588

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title

• G01S17/89 Primary

  for mapping or imaging · CPC title

• G06F18/253

  of extracted features · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title