System and method for identifying travel way features for autonomous vehicle motion control

Patent metadata

Field	Value
Publication number	US-11217012-B2
Application number	US-201916684689-A
Country	US
Kind code	B2
Filing date	Nov 15, 2019
Priority date	Nov 16, 2018
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

Systems and methods for identifying travel way features in real time are provided. A method can include receiving two-dimensional and three-dimensional data associated with the surrounding environment of a vehicle. The method can include providing the two-dimensional data as one or more input into a machine-learned segmentation model to output a two-dimensional segmentation. The method can include fusing the two-dimensional segmentation with the three-dimensional data to generate a three-dimensional segmentation. The method can include storing the three-dimensional segmentation in a classification database with data indicative of one or more previously generated three-dimensional segmentations. The method can include providing one or more datapoint sets from the classification database as one or more inputs into a machine-learned enhancing model to obtain an enhanced three-dimensional segmentation. And, the method can include identifying one or more travel way features based at least in part on the enhanced three-dimensional segmentation.

First claim

What is claimed is:

1. A computer-implemented method for identifying travel way features, the method comprising:
obtaining two-dimensional data depicting a surrounding environment of a vehicle and three-dimensional data associated with the surrounding environment;
providing the two-dimensional data as one or more inputs into a machine-learned segmentation model to receive a two-dimensional segmentation as one or more outputs of the machine-learned segmentation model;
generating a current three-dimensional segmentation based at least in part on the three-dimensional data and the two-dimensional segmentation;
storing first data indicative of the current three-dimensional segmentation in a classification database, wherein the classification database comprises second data associated with one or more previously generated three-dimensional segmentations, wherein the second data comprises data indicative of a plurality of representations of historical three-dimensional datapoints associated with the one or more previously generated three-dimensional segmentations, wherein each historical three-dimensional datapoint in the plurality of representations of historical three-dimensional datapoints comprises a three-dimensional point, wherein the second data comprises a respective three-dimensional point for each historical three-dimensional datapoint in the plurality of representations of historical three-dimensional datapoints and a feature vector corresponding to each three-dimensional point; and
identifying, based on data stored in the classification database, one or more travel way features.

2. The computer-implemented method of claim 1 , wherein the classification database comprises a non-parametric memory configured to maintain a record of three-dimensional datapoints corresponding to the one or more travel way features.

3. The computer-implemented method of claim 1 , wherein identifying the one or more travel way features based on data stored in the classification database comprises:
identifying a plurality of datapoint sets from the classification database, wherein each of the one or more datapoint sets comprises one or more three-dimensional points and at least one feature vector corresponding to each of the one or more three-dimensional points;
providing each of the plurality of datapoint sets as an input into a machine-learned enhancing model to receive an enhanced three-dimensional segmentation; and
identifying the one or more travel way features based at least in part on the enhanced three-dimensional segmentation.

4. The computer-implemented method of claim 1 , wherein the feature vector corresponding to each three-dimensional point comprises data indicative of at least one of a feature classification, an occlusion score, a LiDAR intensity, or a vehicle distance corresponding to a respective three-dimensional point.

5. The computer-implemented method of claim 4 , further comprising:
determining the occlusion score for each of the plurality of representations of historical three-dimensional datapoints based at least in part on the current three-dimensional segmentation; and
storing the occlusion score for each of the plurality of representations of historical three-dimensional datapoints in the classification database.

6. The computer-implemented method of claim 1 , wherein the two-dimensional data comprises a plurality of pixels forming an input image, and the three-dimensional data comprises three-dimensional point cloud data indicative of LIDAR data.

7. The computer-implemented method of claim 6 , wherein the two-dimensional segmentation comprises data indicative of one or more classifications associated with one or more of the plurality of pixels forming the input image, and the current three-dimensional segmentation comprises data indicative of a plurality of current three-dimensional datapoints, each of the plurality current three-dimensional datapoints comprising a current three-dimensional point and one or more feature classifications associated with the current three-dimensional point.

8. The computer-implemented method of claim 1 , wherein generating the current three-dimensional segmentation comprises:
fusing the two-dimensional data, the three-dimensional data and the two-dimensional segmentation.

9. A computing system comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations comprising:
obtaining two-dimensional data depicting a surrounding environment of a vehicle and three-dimensional data associated with the surrounding environment;
providing the two-dimensional data as one or more inputs into a machine-learned segmentation model to receive a two-dimensional segmentation as one or more outputs of the machine-learned segmentation model;
generating a current three-dimensional segmentation based at least in part on the three-dimensional data and the two-dimensional segmentation;
storing data indicative of the current three-dimensional segmentation in a classification database, wherein the classification database comprises second data associated with one or more previously generated three-dimensional segmentations, wherein the second data comprises data indicative of a plurality of representations of historical three-dimensional datapoints associated with the one or more previously generated three-dimensional segmentations, wherein each historical three-dimensional datapoint in the plurality of representations of historical three-dimensional datapoints comprises a three-dimensional point, wherein the second data comprises a respective three-dimensional point for each historical three-dimensional datapoint in the plurality of representations of historical three-dimensional datapoints and a feature vector corresponding to each three-dimensional point; and
identifying one or more travel way features based, at least in part, on data stored in the classification database.

10. The computing system of claim 9 , wherein the current three-dimensional segmentation comprises a plurality of current three-dimensional datapoints, each of the plurality of current three-dimensional datapoints comprising a current three-dimensional point and one or more feature classifications associated with the current three-dimensional point, and wherein the current three-dimensional point comprises one or more three-dimensional cartesian coordinates.

11. The computer-implemented method of claim 10 , wherein the classification database defines a dynamic graph based on a spatial relationship between each of a plurality of three-dimensional points.

12. The computing system of claim 11 , wherein the feature vector comprises data indicative of at least one of a feature classification, occlusion score, LiDAR intensity, or vehicle distance corresponding to the three-dimensional point for each historical three-dimensional datapoint in the plurality of representations of historical three-dimensional datapoints.

13. The computing system of claim 12 , wherein identifying the one or more travel way features based at least in part data stored in the classification database comprises:
identifying a plurality of datapoint sets from the classification database, wherein each of the plurality of datapoint sets comprises one or more three-dimensional points and the feature vector corresponding to each of the one or more three-dimensional points;
providing each of the plurality of datapoint sets as an input into a machine-learned enhancing model to receive an enhanced three-dimensional segmentation; and

Assignees

• Uatc Llc

Inventors

• Urtasun Raquel
• Bai Min
• Wang Shenlong

Classifications

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06V20/582

  of traffic signs · CPC title

• G01S17/931 Primary

  of land vehicles · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06V10/803

  of input or preprocessed data · CPC title