Autonomous vehicle lane boundary detection systems and methods

Patent metadata

Field	Value
Publication number	US-11080537-B2
Application number	US-201816122413-A
Country	US
Kind code	B2
Filing date	Sep 5, 2018
Priority date	Nov 15, 2017
Publication date	Aug 3, 2021
Grant date	Aug 3, 2021

Abstract

Systems and methods for facilitating communication with autonomous vehicles are provided. In one example embodiment, a computing system can obtain a first type of sensor data (e.g., camera image data) associated with a surrounding environment of an autonomous vehicle and/or a second type of sensor data (e.g., LIDAR data) associated with the surrounding environment of the autonomous vehicle. The computing system can generate overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the first and/or second types of sensor data. The computing system can determine one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on the overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle and a machine-learned lane boundary detection model.

First claim

What is claimed is:

1. A computing system, comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations comprising:
obtaining a first type of sensor data associated with a surrounding environment of an autonomous vehicle and a second type of sensor data associated with the surrounding environment of the autonomous vehicle;
inputting the second type of sensor data associated with the surrounding environment of the autonomous vehicle into a machine-learned ground height estimation model;
obtaining, from the machine-learned ground height estimation model, a ground height estimation;
generating two-dimensional overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the first type of sensor data and the ground height estimation;
accessing data indicative of a machine-learned lane boundary detection model;
inputting the two-dimensional overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle into the machine-learned lane boundary detection model; and
obtaining an output from the machine-learned lane boundary detection model, wherein the output is indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.

2. The computing system of claim 1 , wherein the first type of sensor data comprises camera image data associated with the surrounding environment of the autonomous vehicle.

3. The computing system of claim 2 , wherein the second type of sensor data comprises LIDAR data associated with the surrounding environment of the autonomous vehicle.

4. The computing system of claim 3 , wherein generating the two-dimensional overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle comprises:
determining the ground height estimation associated with the surrounding environment of the autonomous vehicle based at least in part on the LIDAR data; and
generating the two-dimensional overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle based at least in part on the ground height estimation and the camera image data.

5. The computing system of claim 4 , wherein the method further comprises:
accessing data indicative of the machine-learned ground height estimation model, the machine-learned ground height estimation model being configured to provide a second output indicative of the ground height estimation.

6. The computing system of claim 4 , wherein generating the two-dimensional overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle based at least in part on the ground height estimation and the camera image data comprises:
accessing data indicative of a machine-learned feature projection model;
inputting the camera image data associated with the surrounding environment of the autonomous vehicle into the machine-learned feature projection model; and
obtaining a third output from the machine-learned feature projection model, wherein the third output is indicative a multi-dimensional feature volume of the camera image data.

7. The computing system of claim 6 , wherein generating the two-dimensional overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle based at least in part on the ground height estimation and the camera image data comprises:
generating the overhead image data based at least in part on the multi-dimensional feature volume associated with the camera image data and the ground height estimation.

8. The computing system of claim 1 , wherein the machine-learned lane boundary detection model comprises a convolutional neural network.

9. The computing system of claim 1 , wherein the operations further comprise:
inputting the second type of sensor data associated with the surrounding environment of the autonomous vehicle into the machine-learned lane boundary detection model.

10. A computer-implemented method of detecting lane boundaries, the method comprising:
obtaining, by a computing system comprising one or more computing devices, a first type of sensor data associated with a surrounding environment of an autonomous vehicle;
determining, by the computing system, a multi-dimensional feature volume of the first type of sensor data based at least in part on a first machine-learned model and the first type of sensor data;
obtaining, by the computing system, a second type of sensor data associated with the surrounding environment of the autonomous vehicle;
determining, by the computing system, a ground height estimation associated with the surrounding environment of the autonomous vehicle based at least in part on inputting the second type of sensor data into a machine-learned ground height estimation model and obtaining, from the machine-learned ground height estimation model, the ground height estimation;
generating, by the computing system, overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the multi-dimensional feature volume and the ground height estimation; and
determining, by the computing system, one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on the overhead image data and a third machine-learned model.

11. The computer-implemented method of claim 10 , wherein the first type of sensor data comprises camera image data and the second type of sensor data comprises rasterized LIDAR data.

12. The computer-implemented method of claim 10 , further comprising:
initiating, by the computing system, a performance of one or more vehicle actions by the autonomous vehicle based at least in part on the one or more lane boundaries.

13. The computer-implemented method of claim 12 , wherein the one or more vehicle actions comprise planning a motion of the autonomous vehicle based at least in part on the one or more lane boundaries.

14. The computer-implemented method of claim 12 , wherein the one or more vehicle actions comprise perceiving an object within the surrounding environment of the autonomous vehicle based at least in part on the one or more lane boundaries.

15. The computer-implemented method of claim 12 , wherein the one or more vehicle actions comprise predicting a motion trajectory of an object within the surrounding environment of the autonomous vehicle based at least in part on the one or more lane boundaries.

16. The computer-implemented method of claim 10 , further comprising:
obtaining, by the computing system, a third type of sensor data associated with the autonomous vehicle; and
wherein determining the one or more lane boundaries within the surrounding environment of the autonomous vehicle comprises determining the one or more lane boundaries within the surrounding environment of the autonomous vehicle also based at least in part on the third type of sensor data.

17. An autonomous vehicle comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the autonomous vehicle to perform operations comprising:
obtaining camera image data associated with a surrounding environment of the autonomous vehicle an

Assignees

• Uatc Llc

Inventors

• Bai Min
• Mattyus Gellert Sandor
• Homayounfar Namdar
• Wang Shenlong
• Lakshmikanth Shrindihi Kowshika
• Urtasun Raquel

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06V20/588

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06N3/045

  Combinations of networks · CPC title