Systems and methods for generating synthetic light detection and ranging data via machine learning

Patent metadata

Field	Value
Publication number	US-11461963-B2
Application number	US-201916567607-A
Country	US
Kind code	B2
Filing date	Sep 11, 2019
Priority date	Nov 16, 2018
Publication date	Oct 4, 2022
Grant date	Oct 4, 2022

Abstract

The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

First claim

What is claimed is:

1. A computer-implemented method to generate synthetic light detection and ranging (LiDAR) data, the method comprising:
obtaining, by a computing system comprising one or more computing devices, a three-dimensional map of an environment;
determining, by the computing system, a trajectory that describes a series of locations of a virtual object relative to the environment over time;
performing, by the computing system, a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein a respective depth is associated with a respective point of the plurality of points;
processing, by the computing system using a machine-learned geometry network, the initial three-dimensional point cloud to predict a respective adjusted depth for the respective point; and
generating, by the computing system, an adjusted three-dimensional point cloud in which the respective point has the respective adjusted depth predicted by the machine-learned geometry network.

2. The computer-implemented method of claim 1 , further comprising:
generating, by the computing system, a respective intensity value for the respective point based at least in part on intensity data included in the three-dimensional map for locations within a radius of a respective location associated with the respective point in either the initial three-dimensional point cloud or the adjusted three-dimensional point cloud.

3. The computer-implemented method of claim 1 , wherein performing, by the computing system, the ray casting to generate the initial three-dimensional point cloud comprises determining, by the computing system for a respective ray of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory.

4. The computer-implemented method of claim 3 , wherein performing, by the computing system, the ray casting to generate the initial three-dimensional point cloud comprises:
identifying, by the computing system for the respective ray, a closest surface element in the three-dimensional map to the ray casting location and along the ray casting direction; and
generating, by the computing system for the respective ray, the respective point with the respective depth based at least in part on a distance from the ray casting location to the closest surface element.

5. The computer-implemented method of claim 1 , further comprising feeding, by the computing system, the adjusted three-dimensional point cloud as LiDAR data input to an autonomy computing system of an autonomous vehicle to test a performance of the autonomy computing system of the autonomous vehicle in the environment.

6. The computer-implemented method of claim 1 , wherein the machine-learned geometry network comprises a parametric continuous convolution network.

7. The computer-implemented method of claim 1 , wherein the machine-learned geometry network comprises a plurality of continuous fusion layers with residual connections between adjacent layers.

8. The computer-implemented method of claim 1 , wherein obtaining, by the computing system, the three-dimensional map of the environment comprises generating, by the computing system, the three-dimensional map, and generating, by the computing system, the three-dimensional map comprises:
obtaining, by the computing system, a plurality of sets of real-world LiDAR data physically collected by one or more LiDAR systems in the environment;
removing, by the computing system, one or more moving objects from the plurality of sets of real-world LiDAR data;
associating, by the computing system, the plurality of sets of real-world LiDAR data to a common coordinate system to generate an aggregate LiDAR point cloud; and
converting, by the computing system, the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh.

9. The computer-implemented method of claim 1 , wherein the machine-learned geometry network has been trained using an objective function that comprises a reconstruction loss term that measures respective distances between points included in a synthetic three-dimensional point cloud generated using the machine-learned geometry network and points included in a ground truth three-dimensional point cloud collected by a physical LiDAR system.

10. The computer-implemented method of claim 1 , wherein the machine-learned geometry network has been trained using an objective function that comprises an adversarial loss term that measures an ability of a discriminator network to select which of a synthetic three-dimensional point cloud generated using the machine-learned geometry network and a ground truth three-dimensional point cloud collected by a physical LiDAR system is real and which is synthetic.

11. A computing system, comprising:
one or more processors;
a machine-learned geometry network; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a ground truth three-dimensional point cloud collected by a physical LiDAR system as the physical LiDAR system travelled along a trajectory through an environment;
obtaining a three-dimensional map of the environment;
performing a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein a respective depth is associated with a respective point of the plurality of points;
processing, by the machine-learned geometry network, the initial three-dimensional point cloud to predict a respective adjusted depth for the respective point;
generating an adjusted three-dimensional point cloud in which the respective point has the respective adjusted depth predicted by the machine-learned geometry network;
evaluating an objective function that compares the adjusted three-dimensional point cloud to the ground truth three-dimensional point cloud; and
modifying one or more values of one or more parameters of the machine-learned geometry network based at least in part on the objective function.

12. The computing system of claim 11 , wherein evaluating the objective function comprises determining respective distances between the plurality of points included in the adjusted three-dimensional point cloud and ground truth points included in the ground truth three-dimensional point cloud collected by the physical LiDAR system.

13. The computing system of claim 11 , wherein evaluating the objective function comprises:
providing the adjusted three-dimensional point cloud and the ground truth three-dimensional point cloud to a discriminator network;
receiving, from the discriminator network, a selection of one of the adjusted three-dimensional point cloud and the ground truth three-dimensional point cloud as real; and
determining a loss value based at least in part on the selection received from the discriminator network.

14. The computing system of claim 11 , wherein modifying the one or more values of the one or more parameters of the machine-learned geometry network based at least in part on the objective function comprises backpropagating the objective function through the machine-learned geometry network.

15. The computing system of claim 11 , wherein the machine-learned geometry network comprises a parametric continuous convolution net

Assignees

• Uatc Llc

Inventors

• Manivasagam Sivabalan
• Wang Shenlong
• Ma Wei-Chiu
• Urtasun Raquel

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G01S17/931 Primary

  of land vehicles · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title