Systems and methods for generating synthetic light detection and ranging data via machine learning

Patent metadata

Field	Value
Publication number	US-12106435-B2
Application number	US-202318345431-A
Country	US
Kind code	B2
Filing date	Jun 30, 2023
Priority date	Nov 16, 2018
Publication date	Oct 1, 2024
Grant date	Oct 1, 2024

Abstract

The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned geometry model can predict one or more adjusted depths for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

First claim

What is claimed is:

1. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated using a physics-based simulation engine to obtain an initial point cloud and using a machine-learned simulation model to adjust the initial point cloud to cause the synthetic LiDAR data to appear more realistic; and
instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising:
obtaining LiDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.

2. The autonomous vehicle control system of claim 1 , wherein the at least one of the machine-learned models comprises a machine-learned perception model.

3. The autonomous vehicle control system of claim 2 , wherein the machine-learned perception model was tested by inputting the synthetic LiDAR data to the machine-learned perception model and evaluating an output of the machine-learned perception model.

4. The autonomous vehicle control system of claim 1 , wherein the machine-learned simulation model was trained by evaluating a loss over synthetic point clouds generated using the machine-learned simulation model and ground truth point clouds collected by a physical LiDAR system.

5. The autonomous vehicle control system of claim 4 , wherein the loss is configured to correspond to a perceptual similarity between the synthetic point clouds and the ground truth point clouds.

6. The autonomous vehicle control system of claim 1 , wherein the machine-learned simulation model comprises a machine-learned geometry network configured to adjust a geometry of the initial point cloud.

7. The autonomous vehicle control system of claim 1 , wherein the synthetic LiDAR data comprises a plurality of points descriptive of an object in an environment, wherein the object is a virtual object inserted into the environment.

8. The autonomous vehicle control system of claim 1 , wherein the machine-learned simulation model is configured to correct artifacts in the initial point cloud.

9. One or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated using a physics-based simulation engine to obtain an initial point cloud and using a machine-learned simulation model to adjust the initial point cloud to cause the synthetic LiDAR data to appear more realistic; and
instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations, the operations comprising:
obtaining LiDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LIDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.

10. The one or more non-transitory computer-readable media of claim 9 , wherein the at least one of the machine-learned models comprises a machine-learned perception model.

11. The one or more non-transitory computer-readable media of claim 10 , wherein the machine-learned perception model was tested by inputting the synthetic LiDAR data to the machine-learned perception model and evaluating an output of the machine-learned perception model.

12. The one or more non-transitory computer-readable media of claim 9 , wherein the machine-learned simulation model was trained by evaluating a loss over synthetic point clouds generated using the machine-learned simulation model and ground truth point clouds collected by a physical LiDAR system.

13. The one or more non-transitory computer-readable media of claim 12 , wherein the loss is configured to correspond to a perceptual similarity between the synthetic point clouds and the ground truth point clouds.

14. The one or more non-transitory computer-readable media of claim 9 , wherein the machine-learned simulation model comprises a machine-learned geometry network configured to adjust a geometry of the initial point cloud.

15. The one or more non-transitory computer-readable media of claim 9 , wherein the machine-learned simulation model is configured to correct artifacts in the initial point cloud.

16. A method, comprising:
obtaining light detection and ranging (LiDAR) data descriptive of an environment of an autonomous vehicle;
determining, using one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle, wherein at least one of the one or more machine-learned models was tested using synthetic LiDAR data, wherein the synthetic LiDAR data was generated using a physics-based simulation engine to obtain an initial point cloud and using a machine-learned simulation model to adjust the initial point cloud to cause the synthetic LiDAR data to appear more realistic; and
controlling the autonomous vehicle according to the motion plan.

17. The method of claim 16 , wherein the at least one of the machine-learned models comprises a machine-learned perception model.

18. The method of claim 17 , wherein the machine-learned perception model was tested by inputting the synthetic LiDAR data to the machine-learned perception model and evaluating an output of the machine-learned perception model.

19. The method of claim 16 , wherein the machine-learned simulation model was trained by evaluating a loss over synthetic point clouds generated using the machine-learned simulation model and ground truth point clouds collected by a physical LiDAR system.

20. The method of claim 16 , wherein the machine-learned simulation model comprises a machine-learned geometry network configured to adjust a geometry of the initial point cloud.

21. The method of claim 16 , wherein the machine-learned simulation model is configured to correct artifacts in the initial point cloud.

Assignees

• Aurora Operations Inc

Inventors

• Manivasagam Sivabalan
• Wang Shenlong
• Ma Wei-Chiu
• Urtasun Raquel

Classifications

• G06N3/0475

  Generative networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title