Systems and methods for generating synthetic light detection and ranging data via machine learning

Patent metadata

Field	Value
Publication number	US-11734885-B2
Application number	US-202217958797-A
Country	US
Kind code	B2
Filing date	Oct 3, 2022
Priority date	Nov 16, 2018
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

First claim

What is claimed is:

1. A computer-implemented method to generate synthetic light detection and ranging (LiDAR) data, the method comprising:
generating, using a physics-based simulation engine and based at least in part on an object in an environment, an initial point cloud that comprises a plurality of points descriptive of the object; and
generating, using a machine-learned geometry network and based at least in part on the initial point cloud, an adjusted point cloud,
wherein the machine-learned geometry network was trained by evaluating a loss over synthetic point clouds generated using the machine-learned geometry network and ground truth point clouds collected by a physical LiDAR system, the loss configured to correspond to a perceptual similarity between the synthetic point clouds and the ground truth point clouds.

2. The computer-implemented method of claim 1 , comprising:
inputting the adjusted point cloud to a machine-learned perception system for an autonomous vehicle to simulate real-world LIDAR data; and
evaluating an output of the machine-learned perception system generated based at least in part on the adjusted point cloud.

3. The computer-implemented method of claim 1 , wherein the adjusted point cloud corresponds to a new view of the object.

4. The computer-implemented method of claim 1 , wherein the object is a virtual object inserted into the environment.

5. The computer-implemented method of claim 1 , comprising:
simulating a virtual LIDAR system moving along a trajectory through the environment,
wherein the adjusted point cloud corresponds to a simulated output of the virtual LIDAR system.

6. The computer-implemented method of claim 5 , wherein simulating the virtual LIDAR system comprises determining a ray casting location and a ray casting direction based at least in part on the trajectory, the ray casting location and the ray casting direction being used by the physics-based simulation engine to generate the initial point cloud.

7. The computer-implemented method of claim 1 , comprising:
obtaining real-world LiDAR data of the object physically collected by a LiDAR system in the environment;
converting the real-world LiDAR data to a mesh representation of the object; and
generating the initial point cloud using the mesh representation of the object.

8. A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
generating, using a physics-based simulation engine and based at least in part on an object in an environment, an initial point cloud that comprises a plurality of points descriptive of the object; and
generating, using a machine-learned geometry network and based at least in part on the initial point cloud, an adjusted point cloud,
wherein the machine-learned geometry network was trained by evaluating a loss over synthetic point clouds generated using the machine-learned geometry network and ground truth point clouds collected by a physical LiDAR system, the loss configured to correspond to a perceptual similarity between the synthetic point clouds and the ground truth point clouds.

9. The computing system of claim 8 , wherein the operations further comprise:
inputting the adjusted point cloud to a machine-learned perception system for an autonomous vehicle to simulate real-world LIDAR data; and
evaluating an output of the machine-learned perception system generated based at least in part on the adjusted point cloud.

10. The computing system of claim 8 , wherein the adjusted point cloud corresponds to a new view of the object.

11. The computing system of claim 8 , wherein the object is a virtual object inserted into the environment.

12. The computing system of claim 11 , wherein the virtual object is a virtual vehicle inserted into the environment.

13. The computing system of claim 8 , wherein the operations further comprise:
simulating a virtual LIDAR system moving along a trajectory through the environment,
wherein the adjusted point cloud corresponds to a simulated output of the virtual LIDAR system.

14. The computing system of claim 13 , wherein simulating the virtual LIDAR system comprises determining a ray casting location and a ray casting direction based at least in part on the trajectory, the ray casting location and the ray casting direction being used by the physics-based simulation engine to generate the initial point cloud.

15. The computing system of claim 8 , wherein the operations further comprise:
obtaining real-world LiDAR data of the object physically collected by a LiDAR system in the environment;
converting the real-world LiDAR data to a mesh representation of the object; and
generating the initial point cloud using the mesh representation of the object.

16. The computing system of claim 8 , wherein the computing system is onboard an autonomous vehicle.

17. The computing system of claim 16 , wherein the autonomous vehicle is an autonomous truck.

18. A system for training geometry models for generating synthetic light detection and ranging (LiDAR) data, the system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
generating, using a physics-based simulation engine and based at least in part on an object in an environment, an initial point cloud that comprises a plurality of points descriptive of the object; and
generating, using a machine-learned geometry network and based at least in part on the initial point cloud, an adjusted point cloud;
evaluating a loss over the adjusted point cloud and a corresponding ground truth point cloud collected by a physical LiDAR system, the loss configured to correspond to a perceptual similarity between the adjusted point cloud and the corresponding ground truth point cloud collected; and
updating one or more parameters of the machine-learned geometry network based at least in part on the loss.

19. The system of claim 18 , wherein the operations further comprise:
obtaining real-world LiDAR data of the object physically collected by a LiDAR system in the environment;
converting the real-world LiDAR data to a mesh representation of the object; and
generating the initial point cloud using the mesh representation of the object.

20. The system of claim 18 , wherein the object is a virtual object inserted into the environment.

Assignees

• Uatc Llc

Inventors

• Manivasagam Sivabalan
• Wang Shenlong
• Ma Wei-Chiu
• Urtasun Raquel

Classifications

• G06N3/0475

  Generative networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title