Verification module system and method for motion-based lane detection with multiple sensors

Patent metadata

Field	Value
Publication number	US-11573095-B2
Application number	US-202017074468-A
Country	US
Kind code	B2
Filing date	Oct 19, 2020
Priority date	Aug 22, 2017
Publication date	Feb 7, 2023
Grant date	Feb 7, 2023

Abstract

A method of lane detection for a non-transitory computer readable storage medium storing one or more programs is disclosed. The one or more programs include instructions, which when executed by a computing device, cause the computing device to perform the following steps comprising: generating a ground truth associated with lane markings expressed in god's view; receiving features from at least one of a hit-map image and a fitted lane marking, wherein the hit-map image includes a classification of pixels that hit a lane marking, and the fitted lane marking includes pixels optimized based on the hit-map image; and training a confidence module based on the features and the ground truth, the confidence module configured to determine on-line whether a fitted lane marking is reasonable, using parameters that express a lane marking in an arc.

First claim

What is claimed is:

1. A method of lane detection, comprising:
receiving images of a road comprising images of at least one lane marking;
receiving a hit-map image associated with a current view of the road, wherein the hit-map image identifies pixels in the hit-map image that hit the at least one lane marking;
receiving a first lane template associated with a previous view of the road previous to the current view of the road;
generating a fitted lane marking using the first lane template and the hit-map image; and
determine a confidence level of the fitted lane marking using parameters of an arc of a circle fitted into the fitted lane marking.

2. The method according to claim 1 , further comprising:
generating a ground truth by annotating the received images to identify the at least one lane marking.

3. The method according to claim 2 , wherein the generating the ground truth comprises:
generating labeled lane markings by annotating the at least one lane marking using arcs of circles.

4. The method according to claim 2 , wherein the ground truth is used to train a lane detection algorithm, the method further comprising:
training the lane detection algorithm relating the images to the hit-map images using the ground truth; and
generating another hit-map image for the current view of the road based on the lane detection algorithm.

5. The method according to claim 1 , wherein the generating the fitted lane marking comprises:
adjusting the first lane template associated with the previous view of the road.

6. The method according to claim 1 , further comprising:
determining that the confidence level of the fitted lane marking is above or equal to a threshold confidence value; and
outputting the fitted lane marking as a predicted lane marking.

7. The method according to claim 6 , further comprising:
generating a ground truth by annotating the received images to identify the at least one lane marking;
comparing the fitted lane marking having the confidence level above or equal to the threshold confidence value to the ground truth; and
determining a failure pattern associated with the fitted lane marking when the comparing indicates that the fitted lane marking fails to match the ground truth.

8. The method according to claim 1 , further comprising:
determining that the confidence level of the fitted lane marking is below a threshold confidence value; and
rejecting the fitted lane marking due to the determining that the confidence level of the fitted lane marking is below the threshold confidence value.

9. The method according to claim 8 , further comprising:
generating a ground truth by annotating the received images to identify the at least one lane marking;
comparing the fitted lane marking having the confidence level below the threshold confidence value to the ground truth; and
determining a failure pattern associated with the fitted lane marking when the comparing indicates that the fitted lane marking matches the ground truth.

10. The method according to claim 1 , further comprising:
generating a ground truth by annotating the received images to identify the at least one lane marking; and
training a confidence module based on the ground truth and at least one of the hit-map image or the fitted lane marking.

11. The method according to claim 10 , further comprising:
refining the confidence module by adding labeled data in training the confidence module.

12. A system for lane detection, the system comprising:
a memory;
one or more processing units; and
one or more programs stored in the memory and configured for execution by the one or more processing units, the one or more programs comprising instructions to cause at least:
receiving images of a road comprising at least one lane marking;
receiving a hit-map image associated with a current view of the road, wherein the hit-map image identifies pixels in the hit-map image that hit the at least one lane marking;
receiving a first lane template associated with a previous view of the road previous to the current view of the road;
generating a fitted lane marking using the first lane template and the hit-map image; and
determining a confidence level of the fitted lane marking using parameters of an arc of a circle fitted into the fitted lane marking.

13. The system according to claim 12 , wherein the one or more programs comprises instructions to further cause at least:
generating a second lane template associated with the current view of the road using the parameters; and
using the second lane template for processing of a next view of the road subsequent to the current view of the road.

14. The system according to claim 12 , wherein the fitted lane marking is represented in a coordinate system in which a vehicle is taken as an origin, a direction from a back of the vehicle to a front of the vehicle is a y-coordinate axis, and a direction from one side of the vehicle to another side of the vehicle is a x-coordinate axis, and wherein the x-coordinate axis is perpendicular to the y-coordinate axis.

15. The system according to claim 12 , wherein the generating the fitted lane marking comprises:
adjusting the first lane template associated with the previous view of the road according to at least one of information of a global positioning system (GPS), an inertial measurement unit (IMU), and a mapping.

16. The system according to claim 12 , wherein the at least one lane marking has a plurality of lane markings, wherein the generating the fitted lane marking further comprises using at least one of the following constraints:
the lane markings are parallel to each other;
a shape of curvatures of lanes is a circle;
a curvature of lanes is smaller than approximately three hundred meters;
a lane spacing between neighboring lanes ranges between approximately three and four meters; and
a color at an edge of a lane marking is different from that at other portions of the road free from the lane marking.

17. The system according to claim 12 , wherein the one or more programs comprises instructions to further cause at least:
generating a ground truth by annotating the received images to identify the at least one lane marking;
performing comparison of the fitted lane marking to the ground truth; and
determining a failure pattern associated with the fitted lane marking when the comparison indicates that the fitted lane marking fails to match the ground truth.

18. A method of lane detection for a non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, cause the computing device to perform the following steps comprising:
receiving images of a road comprising at least one lane marking;
receiving a hit-map image associated with a current view of the road, wherein the hit-map image identifies pixels in the hit-map image that hit the at least one lane marking;
receiving a first lane template associated with a previous view of the road previous to the current view of the road;
generating a fitted lane marking using the first lane template and the hit-map image; and
determine a confidence level of the fitted lane marking using parameters of an arc of a circle fitted into the fitted lane marking.

19. The method according to claim 18 , comprising:
training a confidence module based on the ground truth and at least one of the hit-map image or the fitted lane marking, wherein the ground truth is associated with the at least one lan

Assignees

• Tusimple Inc

Inventors

• Liu Siyuan
• Wang Mingdong
• Hou Xiaodi

Classifications

• G01S17/86

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01C21/3658 Primary

  Lane guidance · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G06V2201/10

  Recognition assisted with metadata · CPC title

• G06V20/588 Primary

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title