Range-view LIDAR-based object detection

Patent metadata

Field	Value
Publication number	US-10310087-B2
Application number	US-201715609256-A
Country	US
Kind code	B2
Filing date	May 31, 2017
Priority date	May 31, 2017
Publication date	Jun 4, 2019
Grant date	Jun 4, 2019

Abstract

Systems and methods for detecting and classifying objects that are proximate to an autonomous vehicle can include receiving, by one or more computing devices, LIDAR data from one or more LIDAR sensors configured to transmit ranging signals relative to an autonomous vehicle, generating, by the one or more computing devices, a data matrix comprising a plurality of data channels based at least in part on the LIDAR data, and inputting the data matrix to a machine-learned model. A class prediction for each of one or more different portions of the data matrix and/or a properties estimation associated with each class prediction generated for the data matrix can be received as an output of the machine-learned model. One or more object segments can be generated based at least in part on the class predictions and properties estimations. The one or more object segments can be provided to an object classification and tracking application.

First claim

What is claimed is:

1. A computer-implemented method of detecting objects of interest comprising:
receiving, by a computing system comprising one or more computing devices, LIDAR data comprising a plurality of LIDAR data points from one or more LIDAR sensors configured to transmit ranging signals relative to an autonomous vehicle;
generating, by the computing system, a data matrix comprising a plurality of data channels based, at least in part, on the LIDAR data, wherein at least one of the plurality of data channels within the data matrix comprises LIDAR Background Subtraction foreground data indicative of whether each of the plurality of LIDAR data points is a foreground LIDAR data point remaining after LIDAR Background Subtraction is applied to the LIDAR data received from the one or more LIDAR sensors;
inputting, by the computing system, the data matrix comprising a plurality of data channels to a machine-learned model;
receiving, by the computing system as a first output of the machine-learned model, a class prediction for each of one or more different portions of the data matrix;
receiving, by the computing system as a second output of the machine-learned model, a properties estimation associated with each class prediction generated for the data matrix;
generating, by the computing system, one or more object segments based at least in part on the class predictions and properties estimations; and
providing, by the computing system, the one or more object segments to an object classification and tracking application.

2. The computer-implemented method of claim 1 , wherein the LIDAR data comprises LIDAR sweep data corresponding to LIDAR point data received around an approximately 360 degree horizontal view around the autonomous vehicle, and wherein the method further comprises generating, by the computing system, the LIDAR sweep data based on the LIDAR data received from the one or more LIDAR sensors.

3. The computer-implemented method of claim 1 , wherein at least one of the plurality of data channels within the data matrix comprises LIDAR point range data indicative of how far each of the plurality of LIDAR data points is from the one or more LIDAR sensors, and wherein at least another one of the plurality of data channels within the data matrix comprises LIDAR point height data indicative of a height above ground for each of the plurality of LIDAR data points.

4. The computer-implemented method of claim 1 , wherein at least one of the plurality of data channels within the data matrix comprises intensity data indicative of an energy intensity of a returned ranging signal received back at the one or more LIDAR sensors after the one or more LIDAR sensors transmits the ranging signals relative to the autonomous vehicle.

5. The computer-implemented method of claim 1 , wherein at least one of the plurality of data channels within the data matrix comprises absence of LIDAR return data indicative of data matrix cells for which no ranging signal was returned after the one or more LIDAR sensors transmits the ranging signals relative to the autonomous vehicle.

6. The computer-implemented method of claim 1 , wherein the data matrix comprises at least five data channels comprising LIDAR point range data indicative of how far each of the plurality of LIDAR data points is from the one or more LIDAR sensors, LIDAR point height data indicative of a height above ground of each of the plurality of LIDAR data points, intensity data indicative of an energy intensity of a returned ranging signal received back at the one or more LIDAR sensors after transmission, absence of LIDAR return data indicative of data matrix cells for which no ranging signal was returned after transmission by the one or more LIDAR sensors, and the LIDAR Background Subtraction foreground data indicative of whether each of the plurality of LIDAR data points is a foreground LIDAR data point remaining after LIDAR Background Subtraction is applied to the LIDAR data from the one or more LIDAR sensors.

7. The computer-implemented method of claim 1 , wherein the machine-learned model comprises a convolutional neural network.

8. The computer-implemented method of claim 1 , the method further comprising:
predicting, by the computing system, an instance segmentation for each of one or more detected instances based at least in part on the class predictions and the properties estimations; and
generating, by the computing system, a bounding box estimation for each instance segmentation based at least in part on the class predictions and the properties estimations.

9. The computer-implemented method of claim 8 , wherein predicting the instance segmentation comprises predicting the instance segmentation based at least in part on a class probability and an instance center determined based on the class predictions and the properties estimations; and
wherein generating the bounding box estimation comprises generating the bounding box estimation based at least in part on an instance center, an orientation, a width, and a height determined based at least in part on the properties estimations.

10. An object detection system comprising:
one or more processors;
a machine-learned prediction model, wherein the prediction model has been trained to receive a data matrix comprising multiple channels of LIDAR-associated data and, in response to receipt of the data matrix comprising multiple channels of LIDAR-associated data, output one or more class predictions for different portions of the data matrix; and
at least one tangible, non-transitory computer readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
obtaining a data matrix comprising multiple channels of LIDAR-associated data, wherein at least one of the multiple channels of LIDAR-associated data comprises LIDAR Background Subtraction foreground data indicative of whether a LIDAR data point is a foreground LIDAR data point remaining after LIDAR Background Subtraction is applied to the LIDAR data received from the one or more LIDAR sensors;
inputting the data matrix comprising multiple channels of LIDAR-associated data into the machine-learned prediction model; and
receiving, as output of the machine-learned prediction model, one or more class predictions for one or more different portions of the data matrix.

11. The object detection system of claim 10 , wherein the machine-learned prediction model has been further trained to output a properties estimation for one or more types of predicted classes identified from the one or more class predictions for the one or more different portions of the data matrix, and
wherein the operations further comprise receiving, as an output of the machine-learned prediction model, a properties estimation associated with each of the one or more class predictions for the one or more different portions of the data matrix.

12. The object detection system of claim 11 , wherein the operations further comprise generating one or more object segments based at least in part on the class predictions and the properties estimations.

13. The object detection system of claim 10 , wherein the output of the machine-learned prediction model includes a class prediction for each cell of the data matrix comprising LIDAR point data.

14. The object detection system of claim 10 , wherein the machine-learned prediction model comprises a convolutional neural network.

15. The object detection system of claim 10 , wherein the operations further comprise:
predicting an instance segmentation for each o

Assignees

• Uber Technologies Inc

Inventors

• Laddha Ankit
• Bagnell J Andrew
• Ramakrishna Varun
• Wang Yimu
• Vallespi-Gonzalez Carlos

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title

• G01S17/50

  Systems of measurement based on relative movement of target · CPC title

• G01S7/4808

  Evaluating distance, position or velocity data · CPC title