Deep learning continuous lane lines detection system for autonomous vehicles

Patent metadata

Field	Value
Publication number	US-11068724-B2
Application number	US-201816158256-A
Country	US
Kind code	B2
Filing date	Oct 11, 2018
Priority date	Oct 11, 2018
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

According to an embodiment, a system receives a captured image perceiving one or more lane markers surrounding the ADV from an image capturing device of the ADV. The system detects one or more continuous lane lines based on the one or more lane markers in the captured image by applying a machine learning model to the captured image, where the machine learning model includes a number of layers of nodes and the machine learning model includes a weighted softmax cross-entropy loss within at least one of the layers in training. The system generates a trajectory based on the one or more continuous (e.g., whole) lane lines to control the ADV autonomously according to the trajectory.

First claim

What is claimed is:

1. A computer-implemented method to generate a driving trajectory for an autonomous driving vehicle (ADV), the method comprising:
receiving a captured image perceiving one or more segmented lane markers surrounding the ADV from an image capturing device of the ADV;
generating one or more continuous lane lines in the captured image by connecting the one or more segmented lane markers in the captured image;
identifying the one or more continuous lane lines representing boundaries of lanes having the segmented lane markers in the captured image by applying a machine learning model to the captured image, wherein the machine learning model includes a plurality of layers of nodes and the machine learning model includes a weighted softmax with cross-entropy loss within at least one of the layers in training, wherein the weighted softmax with cross-entropy is at an output layer of the machine learning model and includes a weighting function determined based on a ratio of a pixels count of the one or more continuous lane lines to a total pixels count of the image, and wherein a weight of the weighting function is unique for each image; and
generating a trajectory based on the one or more continuous lane lines to control the ADV autonomously according to the trajectory.

2. The method of claim 1 , wherein the machine learning model includes a convolutional neural network (CNN) model.

3. The method of claim 2 , wherein the CNN model includes a plurality of convolutional layers and a plurality of deconvolutional layers.

4. The method of claim 3 , wherein the CNN model includes a number of down-sampling and up-sampling layers and the number of down-sampling layers equals the number of up-sampling layers.

5. The method of claim 1 , wherein the segmented lane markers in the captured image comprises dashed, dot, raised, or reflective lane markers in the captured image.

6. The method of claim 1 , further comprising:
determining regions of the captured image having lane markers;
determining a number of whole lane lines corresponding to the regions of the captured image having lane markers;
connecting the lane markers to complete each of the whole lane lines; and
determining the pixels count in the captured image for the completed whole lane lines.

7. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
receiving a captured image perceiving one or more segmented lane markers surrounding an autonomous driving vehicle (ADV) from an image capturing device of the ADV;
generating one or more continuous lane lines in the captured image by connecting the one or more segmented lane markers in the captured image;
identifying the one or more continuous lane lines representing boundaries of lanes having the segmented lane markers in the captured image by applying a machine learning model to the captured image, wherein the machine learning model includes a plurality of layers of nodes and the machine learning model includes a weighted softmax cross-entropy loss within at least one of the layers in training, wherein the weighted softmax with cross-entropy is at an output layer of the machine learning model and includes a weighting function determined based on a ratio of a pixels count of the one or more continuous lane lines to a total pixels count of the image, and wherein a weight of the weighting function is unique for each image; and
generating a trajectory based on the one or more continuous lane lines to control the ADV autonomously according to the trajectory.

8. The non-transitory machine-readable medium of claim 7 , wherein the machine learning model includes a convolutional neural network (CNN) model.

9. The non-transitory machine-readable medium of claim 8 , wherein the CNN model includes a plurality of convolutional layers and a plurality of deconvolutional layers.

10. The non-transitory machine-readable medium of claim 9 , wherein the CNN model includes a number of down-sampling and up-sampling layers and the number of down-sampling layers equals the number of up-sampling layers.

11. The non-transitory machine-readable medium of claim 7 , wherein the segmented lane markers in the captured image comprises dashed, dot, raised, or reflective lane markers in the captured image.

12. The non-transitory machine-readable medium of claim 7 , the operations further comprising:
determining regions of the captured image having lane markers;
determining a number of complete lane lines corresponding to the regions of the captured image having lane markers;
connecting the lane markers to complete each of the whole lane lines; and determining the pixels count in the captured image for the completed whole lane lines.

13. A computer-implemented method to train a machine learning model for an autonomous driving vehicle (ADV), the method comprising:
selecting a set of training images to train the machine learning model, wherein the machine learning model includes a plurality of layers of nodes and a weighted softmax cross-entropy loss within at least one of the layers;
for each of the training images,
identifying one or more continuous lane lines based on one or more lane markers in the image;
generating a plurality of labels by connecting the lane markers corresponding to the identified continuous lane lines;
determining a ratio of pixels count for pixels of the continuous lane lines to a total pixel count of the image;
determining a weighting function for the weighted softmax cross-entropy loss based on the determined ratio; and

training a machine learning model based on the set of training images using the generated labels and the weighted softmax cross-entropy loss, wherein the trained machine learning model is applied to an image perceiving an environment surrounding the ADV captured by an image capturing device of the ADV to identify one or more continuous lane lines based on the lane markers captured in the image, wherein the continuous lane lines are used to generate a trajectory to control the ADV autonomously according to the trajectory.

14. The method of claim 13 , wherein the machine learning model includes a convolutional neural network (CNN) model.

15. The method of claim 14 , wherein the CNN model includes a plurality of convolutional layers and a plurality of deconvolutional layers.

16. The method of claim 15 , wherein the CNN model includes a number of down-sampling and up-sampling layers and the number of down-sampling layers equals the number of up-sampling layers.

17. The method of claim 13 , wherein the weighted softmax cross-entropy loss is at an output layer of the machine learning model.

18. The method of claim 17 , wherein a weight of the weighting function is unique for each training image.

19. The method of claim 18 , wherein the weight of the weighting function is determined based on a ratio of a pixels count of continuous lane markers to a total pixels count of each training image.

20. The method of claim 13 , wherein at least one of the continuous lane lines comprises dashed, dot, raised, or reflective lane markers in the captured image.

Assignees

• Baidu Usa Llc

Inventors

• Liu Xiaoshu

Classifications

• B60W30/12 Primary

  Lane keeping · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/588

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title