System and method for free space estimation

Patent metadata

Field	Value
Publication number	US-11780465-B2
Application number	US-202217951331-A
Country	US
Kind code	B2
Filing date	Sep 23, 2022
Priority date	Jul 10, 2019
Publication date	Oct 10, 2023
Grant date	Oct 10, 2023

Abstract

A system and method for estimating free space including applying a machine learning model to camera images of a navigation area, where the navigation area is broken into cells, synchronizing point cloud data from the navigation area with the processed camera images, and associating probabilities that the cell is occupied and object classifications of objects that could occupy the cells with cells in the navigation area based on sensor data, sensor noise, and the machine learning model.

First claim

What is claimed is:

1. A method for estimating free space based on image data and point cloud data, the free space used for navigating an autonomous vehicle, the method comprising:
semantically classifying the image data based on a machine learning model forming point classifications and point classification probabilities;
associating each point in the point cloud data to the image data that are spatially co-located with the point cloud data;
classifying each of the points as an obstructed space or a non-obstructed space based on the spatial association of each of the points with the semantically classified image data forming obstructed points and non-obstructed points;
forming a grid of the obstructed points and non-obstructed points within a pre-selected area surrounding the autonomous vehicle; and
estimating the free space in the pre-selected area by associating the obstructed points with a first probability based at least on (1) noise in the point cloud data, (2) a second probability that the point cloud data are reliable, (3) a distance from the non-obstructed points to the obstructed space closest to the non-obstructed points, (4) a third probability that the point classifications are correct, and (5) presence of the non-obstructed space.

2. The method as in claim 1 further comprising:
receiving the image data and the point cloud data into the autonomous vehicle.

3. The method as in claim 1 further comprising:
performing a first transform on the points in the point cloud data into an image coordinate system associated with the image data.

4. The method as in claim 3 further comprising:
performing a second transform on the first transformed points into a robot coordinate system associated with the autonomous vehicle.

5. The method as in claim 1 wherein the image data comprise streaming data from a pre-selected number of at least one camera, the at least one camera providing a 360° view of an area surrounding the autonomous vehicle.

6. The method as in claim 1 wherein the machine learning model comprises an ICNET semantic segmentation model.

7. The method as in claim 1 wherein the machine learning model comprises detecting at least one drivable surface.

8. The method as in claim 7 wherein the at least one drivable surface is selected from a group consisting of road, sidewalk, ground, terrain surfaces, and lane markings.

9. The method as in claim 1 wherein the associating each point comprises:
receiving the image data at a pre-selected rate; and
mapping the point cloud data onto the image data.

10. A system for estimating free space based on image data and point cloud data, the free space used for navigating an autonomous vehicle, the autonomous vehicle having a front, a left side, and a right side, the system comprising:
a pre-processor configured to receive camera image data from at least one camera, the pre-processor configured to semantically classify each pixel of the image data into a classification and calculate a probability associated with the classification, the classification and the probability being determined by a machine learning model; and
a free space estimator configured to compute an obstacle classification layer, a probability layer, and a log odds layer based on the point cloud data and the image data;
wherein the free space estimator comprises:
a 3D point cloud to 3D image processor configured to transform the 3D point cloud to 3D image coordinates;
a 3D image to 2D RGB transform configured to transform the 3D image coordinates to 2D RGB coordinates; and
a 2D to 3D baselink transform configured to transform the 2D RGB coordinates to 3D baselink coordinates forming transformed point cloud data.

11. The system as in claim 10 wherein the camera image data comprises streaming data from the at least one camera, the at least one camera providing a 360° view of an area surrounding the autonomous vehicle.

12. The system as in claim 10 wherein a number of the at least one camera comprises three cameras.

13. The system as in claim 10 wherein the machine learning model is configured to detect drivable surfaces, the drivable surfaces including lane markings.

14. The system as in claim 10 wherein the free space estimator is configured to receive data having time stamps into a synchronizer, the synchronizer time-synchronizing the point cloud data, the transformed point cloud data, and the classification based on the time stamps.

15. The system as in claim 10 wherein the 3D point cloud to 3D image processor comprises:
receiving the point cloud data from at least one LIDAR sensor, the classification and the probability of each point of the point cloud data, and coordinate transforms;
associating each of the points in the point cloud data to the image data that are spatially co-located with the point cloud data; and
performing a first transform on the points in the point cloud data into an image coordinate system associated with the image data.

16. The system as in claim 15 wherein associating each point in synchronized point cloud data with a synchronized image data comprises:
for each of the points (X,Y,Z) in the synchronized point cloud data, calculating an angle that the point subtends with a center of the synchronized point cloud data, the angle indicating a field of view of the at least one camera, the calculating including:
if X>0 and Y>0 then the angle=0°+atan(Y/X) degrees*180/π;
if X<0 and Y>0 then the angle=90°−atan(X/Y) degrees*180/π;
if X<0 and Y<0 then the angle=180°+atan(Y/X) degrees*180/π; and
if X>0 and Y<0 then the angle=270°−atan(X/Y) degrees*180/π;
mapping each of the points onto a semantic segmentation output image including:
if 312°<the angle≤360° or 48°≥the angle>0° then mapping the point onto the semantic segmentation output image derived from the at least one camera located at the front of the autonomous vehicle;
if 48°<the angle<180° then mapping the point onto the semantic segmentation output image derived from the at least one camera located on the left side; and
if 180°<the angle≤312° then mapping the point onto the semantic segmentation output image derived from the at least one camera located on the right side.

17. The system as in claim 10 wherein the 3D image to 2D RGB transform comprises:
identifying each of the points that represents an obstructed space and a non-obstructed space based on a spatial association of the points with the semantically classified image data.

18. The system as in claim 16 wherein the 2D to 3D baselink transform comprises performing a transform on the points into a robot coordinate system associated with the autonomous vehicle.

19. The system as in claim 18 wherein the 3D baselink to grid transform comprises:
flattening robot coordinate system points (Xbl, Ybl, Zbl) to a 2D grid map surrounding the autonomous vehicle, the 2D grid map extending to a pre-selected radius around the autonomous vehicle; and
identifying at least one cell of the 2D grid map as occupied if a semantic segmentation output point (XRGB, YRGB), of the semantic segmentation output image, spatially associated with the at least one cell corresponds to an obstructed space.

20. The system as in claim 19 wherein the semantic segmentation output point (XRGB, YRGB) comprises values including 0=non-drivable, 1=road, 2=sidewalk, 3=terrain, 4=lane marking, >0=drivable, 0=obstructed.

21. The system as in claim 19 further comprising:
a layer processor configured to
classify each of the points that represent

Assignees

• Deka Products Lp

Inventors

• Gummadi Raajitha
• Langenfeld Christopher C
• Slate Michael J
• Principe Christopher J

Classifications

• G05D1/81

  Handing over between on-board automatic and on-board manual control · CPC title

• G05D1/2435

  Extracting 3D information · CPC title

• B60W60/0011 Primary

  involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G05D1/0251

  extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593) · CPC title