System and method for large-scale lane marking detection using multimodal sensor data

Patent metadata

Field	Value
Publication number	US-10657390-B2
Application number	US-201715822689-A
Country	US
Kind code	B2
Filing date	Nov 27, 2017
Priority date	Nov 27, 2017
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

A system and method for large-scale lane marking detection using multimodal sensor data are disclosed. A particular embodiment includes: receiving image data from an image generating device mounted on a vehicle; receiving point cloud data from a distance and intensity measuring device mounted on the vehicle; fusing the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data; and generating a lane marking map from the set of lane marking points.

First claim

What is claimed is:

1. A system comprising:
a data processor; and
a multimodal lane detection module, executable by the data processor, the multimodal lane detection module being configured to perform a multimodal lane detection operation configured to:
receive image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;
receive point cloud data from a distance and intensity measuring device mounted on the vehicle;
fuse the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, the fusion including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and
generate a lane marking map from the set of lane marking points.

2. The system of claim 1 being configured to perform a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

3. The system of claim 2 being configured to train a neural network to perform the semantic segmentation operation.

4. The system of claim 1 wherein the image generating device is one or more cameras.

5. The system of claim 1 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.

6. The system of claim 1 being configured to receive vehicle metrics from a vehicle subsystem.

7. The system of claim 1 being configured to back-project the image data on the terrain map with the terrain map elevation data.

8. The system of claim 1 being further configured to output the lane marking map to a vehicle control subsystem of the vehicle.

9. A method comprising:
receiving image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;
receiving point cloud data from a distance and intensity measuring device mounted on the vehicle;
fusing the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data the fusing including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and
generating a lane marking map from the set of lane marking points.

10. The method of claim 9 including performing a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

11. The method of claim 10 including training a neural network to perform the semantic segmentation operation.

12. The method of claim 9 wherein the image generating device is one or more cameras.

13. The method of claim 9 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.

14. The method of claim 9 including receiving vehicle metrics from a vehicle subsystem.

15. The method of claim 9 including back-projecting the image data on the terrain map with the terrain map elevation data.

16. The method of claim 9 including outputting the lane marking map to a vehicle control subsystem of the vehicle.

17. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:
receive image data from an image generating device mounted on a vehicle, the received image data corresponding to a particular location;
receive point cloud data from a distance and intensity measuring device mounted on the vehicle;
fuse the image data and the point cloud data to produce a set of lane marking points in three-dimensional (3D) space that correlate to the image data and the point cloud data, the fusion including aligning and orienting the image data with a terrain map corresponding to the particular location and using terrain map elevation data to transform the image data to the 3D space; and
generate a lane marking map from the set of lane marking points.

18. The non-transitory machine-useable storage medium of claim 17 being configured to perform a semantic segmentation operation on the received image data to identify and label objects in the image data with object category labels on a per-pixel basis.

19. The non-transitory machine-useable storage medium of claim 17 being further configured to fit piecewise lines for each lane marking object detected in the received image data.

20. The non-transitory machine-useable storage medium of claim 17 wherein the distance and intensity measuring device is one or more laser light detection and ranging (LIDAR) devices.

Assignees

• Tusimple Inc

Inventors

• Mei Xue
• Hou Xiaodi
• Guo Dazhou
• Wei Yujie

Classifications

• G07C5/008

  communicating information to a remotely located station (transmission systems for measured values G08C) · CPC title

• G01C21/3602

  Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/246

  using feature-based methods, e.g. the tracking of corners or segments · CPC title

• G06T7/10

  Segmentation; Edge detection (motion-based segmentation G06T7/215) · CPC title