Autonomous vehicle controlled based upon a lidar data segmentation system

Patent metadata

Field	Value
Publication number	US-11022693-B1
Application number	US-201816054088-A
Country	US
Kind code	B1
Filing date	Aug 3, 2018
Priority date	Aug 3, 2018
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

An autonomous vehicle is described herein. The autonomous vehicle includes a lidar sensor system. The autonomous vehicle additionally includes a computing system that executes a lidar segmentation system, wherein the lidar segmentation system is configured to identify objects that are in proximity to the autonomous vehicle based upon output of the lidar sensor system. The computing system further includes a deep neural network (DNN), where the lidar segmentation system identifies the objects in proximity to the autonomous vehicle based upon output of the DNN.

First claim

What is claimed is:

1. An autonomous vehicle (AV) comprising:
an engine;
a braking system;
a steering system;
a lidar sensor; and
a computing system that is in communication with the engine, the braking system, the steering system, and the lidar sensor, wherein the computing system comprises:
a processor; and
memory that stores instructions that, when executed by the processor, cause the processor to perform acts comprising:
receiving lidar data, the lidar data based upon output of the lidar sensor, the lidar data comprising a plurality of points representative of positions of objects in a driving environment of the AV;
assigning a label to a first point in the points that indicates that the first point is representative of ground cover or vegetation based upon output of a deep neural network (DNN) that is configured to classify points as being representative of ground cover or vegetation;
generating a segmentation of the lidar data based upon the label being assigned to the first point, wherein generating the segmentation is based upon output of a second neural network that is configured to receive at least a portion of the lidar data as input and to output data indicative of whether a second point in the lidar data and a third point in the lidar data are representative of a same object; and
controlling at least one of the engine, the braking system, or the steering system during operation of the AV in the driving environment based upon the segmentation.

2. The AV of claim 1 , wherein the output of the DNN comprises a probability that the first point is representative of vegetation.

3. The AV of claim 2 , wherein the label indicates that the first point is representative of vegetation, wherein assigning the label is based upon the probability exceeding a threshold value.

4. The AV of claim 3 , wherein the threshold value is 75%.

5. The AV of claim 1 , wherein the output of the DNN comprises a probability that the first point is representative of ground cover, wherein the label indicates that the first point is representative of ground cover, wherein assigning the label is based upon the probability exceeding a threshold value.

6. The AV of claim 1 , wherein generating the segmentation of the lidar data comprises assigning group labels to the points in the lidar data, each group label indicating one of a plurality of groups of points, each group of points representative of a different respective object in the driving environment.

7. The AV of claim 1 , wherein generating the segmentation comprises excluding the first point from input to the second neural network based upon the label being assigned to the first point.

8. The AV of claim 1 , the acts further comprising assigning a respective label to each of a first group of points in the points based upon output of the DNN, the labels assigned to the first group of points indicating that the first group of points are representative of vegetation or ground cover in the driving environment, wherein generating the segmentation is based further upon the labels assigned to the first group of points.

9. The AV of claim 8 , wherein generating the segmentation comprises excluding the first point and the first group of points from input to the second neural network based upon the labels being assigned to the first point and the first group of points.

10. The AV of claim 1 , wherein the label assigned to the first point is a first label, wherein generating the segmentation of the lidar data comprises assigning second and third labels to a second point in the points and a third point in the points, respectively, wherein the second and third labels indicate that the second point and the third point are representative of a same object, wherein the second and third labels are assigned to the second and third points based upon the first label being assigned to the first point.

11. A method for controlling operation of an autonomous vehicle (AV), comprising:
receiving lidar data from a lidar sensor system of the AV, the lidar data based upon output of at least one lidar sensor, the lidar data comprising a plurality of points representative of positions of objects in a driving environment of the AV;
assigning a label to a first point in the points based upon output of a deep neural network (DNN) that is configured to output a probability that a point in lidar data is representative of ground cover or vegetation, the label indicating that the first point is representative of ground cover or vegetation in the driving environment;
generating a segmentation of the lidar data based upon the label being assigned to the first point, wherein generating the segmentation is based upon output of a second neural network that is configured to receive at least a portion of the lidar data as input and to output data indicative of whether a second point in the lidar data and a third point in the lidar data are representative of a same object; and
controlling, based upon the segmentation, at least one of an engine of the AV, a braking system of the AV, or a steering system of the AV during operation of the AV in the driving environment.

12. The method of claim 11 , wherein the output of the DNN comprises a probability that the first point is representative of vegetation, and wherein the label indicates that the first point is representative of vegetation, wherein assigning the label is based upon the probability exceeding a threshold value.

13. The method of claim 11 , wherein the output of the DNN comprises a probability that the first point is representative of ground cover.

14. The method of claim 13 , wherein the label indicates that the first point is representative of ground cover, wherein assigning the label is based upon the probability exceeding a threshold value.

15. The method of claim 11 , wherein generating the segmentation of the lidar data comprises assigning group labels to the points in the lidar data, each group label indicating one of a plurality of groups of points, each group of points representative of a different respective object in the driving environment.

16. The method of claim 11 , wherein generating the segmentation comprises excluding the first point from input to the second neural network based upon the label being assigned to the first point.

17. The method of claim 11 , the acts further comprising assigning a respective label to each of a first group of points in the points based upon output of the DNN, the labels assigned to the first group of points indicating that the first group of points are representative of vegetation or ground cover in the driving environment, wherein generating the segmentation is based further upon the labels assigned to the first group of points.

18. The method of claim 17 , wherein generating the segmentation comprises excluding the first point and the first group of points from input to the second neural network based upon the labels being assigned to the first point and the first group of points.

19. The method of claim 11 , wherein generating the segmentation comprises inputting a subset of the points to the second neural network subsequent to assigning the label to the first point, the subset of the points not including the first point.

20. An autonomous vehicle (AV) comprising:
a computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:
receiving a lidar point cloud from a lidar sensor system of the AV, the lidar point cloud based upon output of a

Assignees

• Gm Global Tech Operations Llc

Inventors

• Allais Andrea
• Cadien Adam Samuel
• Branson Elliot
• Xie William Gongshu
• Chambers Micah Christopher

Classifications

• G01S17/931 Primary

  of land vehicles · CPC title

• G01S7/4808 Primary

  Evaluating distance, position or velocity data · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title