Lane detection and tracking techniques for imaging systems

Patent metadata

Field	Value
Publication number	US-12266146-B2
Application number	US-202318340573-A
Country	US
Kind code	B2
Filing date	Jun 23, 2023
Priority date	Jan 6, 2020
Publication date	Apr 1, 2025
Grant date	Apr 1, 2025

Abstract

A system for detecting boundaries of lanes on a road is presented. The system includes an imaging system configured to produce a set of pixels associated with lane markings on a road. The system also includes one or more processors configured to detect boundaries of lanes on the road, including: receive, from the imaging system, the set of pixels associated with lane markings; partition the set of pixels into a plurality of groups, each of the plurality of groups associated with one or more control points; and generate a first spline that traverses the control points of the plurality of groups, the first spline describing a boundary of a lane on the road.

First claim

What is claimed is:

1. A system, comprising:
an imaging system configured to produce a set of pixels associated with lane markings on a road; and
one or more processors configured to detect boundaries of lanes on the road, including:
receive, from the imaging system, the set of pixels associated with lane markings;
partition the set of pixels into a plurality of groups, each of the plurality of groups associated with one or more control points; and
generate a first spline that traverses the control points of the plurality of groups, the first spline describing a boundary of a lane on the road.

2. The system of claim 1 , wherein partitioning the set of pixels into the plurality of groups comprises:
iteratively linking the pixels from the set of pixels into one or more linked sections of pixels; and
partitioning the one or more linked sections of pixels into the plurality of groups, including:
determining a principal component for each group of the plurality of groups, wherein each principal component includes at least one control point of the one or more control points; and
associating groups of the plurality of groups based on the respective principal components.

3. The system of claim 2 , wherein determining the principal component for each group comprises fitting a principal component line to the pixels included in each group, wherein the principal component line is bounded by two control points.

4. The system of claim 3 , wherein the respective two control points corresponding to each principal component define a set of control points, and wherein partitioning the set of pixels into the plurality of groups further comprises executing an association algorithm, the association algorithm comprising:
determining an endpoint distance between respective pairs of control points from two different groups;
comparing the endpoint distance for each respective pair of control points with a minimum threshold endpoint distance; and
responsive to determining that the endpoint distance for a respective pair of control points is less than or equal to the minimum threshold endpoint distance, associating the two groups including the respective pairs of control points.

5. The system of claim 3 , wherein the respective two control points corresponding to each principal component define a set of control points, and wherein partitioning the set of pixels into the plurality of groups further comprises executing a graph-based association algorithm, the association algorithm comprising:
assigning each control point of the set of control points to a node in a graph;
forming an edge between each pair of nodes in the graph;
determining a cost for each edge in the graph;
adding a source node and a target node to the graph; and
determining one or more low-cost paths that traverse the graph from the source node to the target node, wherein the control points along each of the low-cost paths correspond to a lane.

6. The system of claim 1 , wherein partitioning the set of pixels into the plurality of groups comprises executing a linking algorithm, the linking algorithm comprising:
generating a bounding box around a first pixel of the set of pixels;
scanning an area encompassed by the bounding box to locate a second pixel; and
linking the first pixel and the second pixel.

7. The system of claim 1 , wherein partitioning the set of pixels into the plurality of groups comprises executing a partitioning algorithm, the partitioning algorithm comprising:
constructing a line between a start pixel of a linked section of pixels and an end pixel of the linked section of pixels;
determining a distance of each pixel between the start pixel and end pixel from the line;
determining that greater than a threshold number of pixels are located beyond a threshold distance from the line; and
separating, in response to determining that greater than the threshold number of pixels are located beyond the threshold distance, the linked section of pixels into two or more separate linked sections of pixels.

8. The system of claim 1 , wherein generating the first spline comprises executing a spline technique, and wherein the spline technique utilizes only the one or more control points.

9. The system of claim 1 , wherein the one or more processors are further configured to identify the set of pixels associated with lane markings, including:
receive, from the imaging system, an image of a road surface, the image made up of a set of pixels; and
determine, within the set of pixels, a first group of pixels associated with lane markings separating a first pair of lanes and a second group of pixels associated with lane markings separating a second pair of lanes, including, for each of the first and second groups:
identify lane marking pixels that correspond to lane markings,
estimate a left boundary and a right boundary relative to the lane marking pixels, such that the left and the right boundaries are substantially aligned with a direction of a lane indicated by the lane marking pixels, and
associate pixels within the left boundary and the right boundary with the corresponding group.

10. The system of claim 9 , wherein associating pixels within the left boundary and the right boundary includes applying a region growth algorithm to grow clusters initially containing only the lane marking pixels.

11. The system of claim 9 , wherein:
determining the first and second groups of pixels includes applying a machine learning model to the image, the machine learning model configured to predict which pixels in the image will be associated with the first group of pixels or the second group of pixels; and
estimating the left and the right boundaries for each group includes generating respective predictions using the machine learning model.

12. The system of claim 9 , wherein:
the image includes a plurality of rows substantially perpendicular to the orientation of the lanes; and
the one or more processors are further configured to, for a lane marking pixel in a certain row, estimate to which next-row pixel in an adjacent row the pixel is connected, wherein the next-row pixel is (i) immediately below the lane marking pixel, (ii) immediately below and one pixel to the left of the marking pixel, or (iii) immediately below and one pixel to the right of the marking pixel.

13. The system of claim 1 , wherein the one or more processors are further configured to track the lane on the road, including:
generate a predicted spline comprising a predicted extension of the first spline in a direction in which the imaging system is moving, wherein the predicted extension of the first spline is generated based at least in part on a curvature of at least a portion of the first spline.

14. The system of claim 13 , wherein generating the predicted extension of the first spline is further based on map data comprising a shape, a direction, or a curvature of the road.

15. The system of claim 13 , wherein the set of pixels is a first set of pixels generated at a first time for a first portion of the road, and the one or more processors are further configured to:
receive, from the imaging system, a second set of pixels associated with lane markings;
generate an updated spline describing the boundary of the lane; and
compare the predicted spline to the second set of pixels to determine a likelihood score.

16. The system of claim 15 , wherein comparing the predicted spline to the second set of pixels to determine the likelihood score comprises:
convolving respective values of (i) a Gaussian kernel derived from the predicted spline and (ii) a

Assignees

• Luminar Tech Inc

Inventors

• Maheshwari Pranav
• Ramezani Vahid R
• El Houcheimi Ismail
• Englard Benjamin

Classifications

• G05D1/244

  using passive navigation aids external to the vehicle, e.g. markers, reflectors or magnetic means · CPC title

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06V10/765

  using rules for classification or partitioning the feature space · CPC title