Distance estimation to objects and free-space boundaries in autonomous machine applications

Patent metadata

Field	Value
Publication number	US-11170299-B2
Application number	US-202016813306-A
Country	US
Kind code	B2
Filing date	Mar 9, 2020
Priority date	Dec 28, 2018
Publication date	Nov 9, 2021
Grant date	Nov 9, 2021

Abstract

In various examples, a deep neural network (DNN) is trained—using image data alone—to accurately predict distances to objects, obstacles, and/or a detected free-space boundary. The DNN may be trained with ground truth data that is generated using sensor data representative of motion of an ego-vehicle and/or sensor data from any number of depth predicting sensors—such as, without limitation, RADAR sensors, LIDAR sensors, and/or SONAR sensors. The DNN may be trained using two or more loss functions each corresponding to a particular portion of the environment that depth is predicted for, such that—in deployment—more accurate depth estimates for objects, obstacles, and/or the detected free-space boundary are computed by the DNN. In some embodiments, a sampling algorithm may be used to sample depth values corresponding to an input resolution of the DNN from a predicted depth map of the DNN at an output resolution of the DNN.

First claim

What is claimed is:

1. A method comprising:
applying image data representative of a field of view of an image sensor to a deployed neural network;
computing, using the deployed neural network and based at least in part on the image data, a depth map comprising first depth data corresponding to one or more objects in the field of view and second depth data corresponding to a free-space boundary associated with the field of view;
associating the first depth data with the one or more objects and associating the second depth data with the free-space boundary; and
performing one or more operations by an ego-vehicle based at least in part on the first depth data and the second depth data.

2. The method of claim 1 , wherein the neural network is trained using a ground truth depth map representative of one or more depth values, the one or more depth values corresponding to one or more free-space boundaries and being generated using: first data representative of future motion of one or more vehicles through one or more environments, second data representative of a point cloud generated using one or more sensors of the one or more vehicles, and third data representative of one or more locations of free-space boundaries within the one or more environments.

3. The method of claim 2 , wherein the future motion of the one or more vehicles is determined from sensor data generated by one or more vehicle sensors associated with the one or more vehicles as the one or more vehicles traverse the one or more environments.

4. The method of claim 1 , wherein the one or more operations include at least one of path planning, world model management, obstacle or collision avoidance, a control decision, or an advanced driver assistance system (ADAS) operation.

5. The method of claim 1 , wherein the neural network is trained using at least one loss function from a group of loss functions to compare a predicted depth map generated by the neural network for an environment to ground truth depth data of the environment, the group of loss functions comprising at least one of:
a first loss function comparing the predicted depth map to ground truth data corresponding to a depth of one or more objects in the environment;
a second loss function comparing the predicted depth map to ground truth data corresponding to a depth of one or more free-space boundaries in the environment; or
a third loss function comparing the predicted depth map to ground truth data corresponding to portions of the environment not including objects or free-space boundaries.

6. The method of claim 1 , wherein a first loss function is used for training the neural network to predict distances to the objects, a second loss function is used for training the neural network to predict the distances to the free-space boundaries, a third loss function is used for training the neural network to predict distances to any portion of one or more environments, and a fourth loss function is used to compute a weighted total loss using the first loss function, the second loss function, and the third loss function.

7. A method comprising:
receiving image data generated by a camera of a vehicle and representative of a field of view of the camera at a first time;
receiving first sensor data representative of future motion of the vehicle in an environment from the first time to a future time;
modeling a ground plane based at least in part on the first sensor data;
determining an updated location of an updated free-space boundary based at least in part on the ground plane and first data representative of an initial location of an initial free-space boundary;
determining one or more depth values corresponding to the updated free-space boundary based at least in part on second sensor data representative of a LIDAR point cloud and the updated location of the updated free-space boundary;
generating a depth map corresponding to the updated free-space boundary; and
training a machine learning model using the depth map as ground truth data.

8. The method of claim 7 , wherein the ground plane is modeled as a piecewise planar ground plane.

9. The method of claim 7 , wherein the determining the updated location includes, for each first pixel of the image associated with the initial free-space boundary:
casting a ray from the camera to a location in world-space corresponding to the first pixel associated with the initial free-space boundary;
determining an intersection point of the ray with the ground plane; and
based at least in point on the intersection point, determining a second pixel location within the image data associated with the updated free-space boundary.

10. The method of claim 7 , wherein the determining the one or more depth values includes correlating pixel locations within the image associated with the updated free-space boundary with LIDAR-based depth values from the LIDAR point cloud.

11. The method of claim 7 , further comprising: generating a first set of depth data from the depth map, the first set of depth data corresponding to one or more objects represented by the image data; and further training the machine learning model using the first set of depth data as additional ground truth data.

12. The method of claim 11 , wherein a first loss function is used to train the machine learning model using a first set of depth data and a second loss function is used to train the machine learning model using a second set of depth data, the second set of depth data being generated from the depth map and corresponding to a depth of one or more free-space boundaries represented by the image data.

13. The method of claim 12 , further comprising: generating a third set of depth data, the third set of depth data corresponding to a portion of the field of view, the generating including correlating the LIDAR point cloud with pixels of the image corresponding to the portion of the environment; and further training the machine learning model using the third set of depth data as additional ground truth data.

14. The method of claim 13 , wherein a third loss function is used to train the machine learning model using the third set of depth data.

15. The method of claim 13 , wherein the generating the third set of depth data includes filtering out portions of the LIDAR point cloud that correspond to at least one of the initial free-space boundary, the updated free-space boundary, or one or more objects depicted in the image.

16. A method comprising: generating a ground truth depth map corresponding to depth values associated with an image of a first spatial resolution; applying image data representative of the image to a neural network; computing, using the neural network, a predicted depth map at a second spatial resolution different from the first spatial resolution; determining, for at least one point in the predicted depth map having an associated first depth value, one or more corresponding neighbor points in the ground truth depth map; based at least in part on a first location of the at least one point, second locations of the one or more neighbor points, and the associated first depth value, executing a sampling algorithm to determine associated second depth values corresponding to each of the neighbor points; and training the neural network based at least in part on a comparison between the associated second depth values and ground truth depth values corresponding to the neighbor points in the ground truth depth map.

17. The method of claim 16 , wherein the determining the neighbor points includes: projecting the point to its corresponding location in the gr

Assignees

• Nvidia Corp

Inventors

• Kwon Junghyun
• Yang Yilin
• Jujjavarapu Bala Siva Sashank
• Ye Zhaoting
• Oh Sangmin
• Park Minwoo
• Nister David

Classifications

• G06V10/763

  Non-hierarchical techniques, e.g. based on statistics of modelling distributions · CPC title

• G06F18/23213

  with fixed number of clusters, e.g. K-means clustering · CPC title

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title