Systems and methods for generating synthetic sensor data via machine learning

Patent metadata

Field	Value
Publication number	US-12222832-B2
Application number	US-202318466286-A
Country	US
Kind code	B2
Filing date	Sep 13, 2023
Priority date	Mar 23, 2019
Publication date	Feb 11, 2025
Grant date	Feb 11, 2025

Abstract

The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

First claim

What is claimed is:

1. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:
using a physics-based simulation engine to obtain an initial synthetic point cloud,
generating, by a machine-learned model, a value corresponding to a probability that a real LIDAR point cloud would have an error at a point in the initial synthetic point cloud, and
generating, based on a determination of whether, based on the value, to include the error in the synthetic LIDAR data, the synthetic LIDAR data to contain the error at the point; and

instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising:
obtaining LiDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.

2. The autonomous vehicle control system of claim 1 , wherein generating, based on the value, the synthetic LIDAR data to contain the error at the point comprises:
sampling, based on the value, to determine whether to include the error at the point in the synthetic LIDAR data.

3. The autonomous vehicle control system of claim 1 , wherein the probability corresponds to a dropout probability.

4. The autonomous vehicle control system of claim 3 , wherein generating the synthetic LIDAR data to contain the error at the point comprises omitting the point.

5. The autonomous vehicle control system of claim 1 , wherein the synthetic LIDAR data was generated by:
generating, using the machine-learned model, a probability mask over the initial synthetic point cloud, the probability mask indicating respective probabilities that a real LiDAR point cloud would have an error at respective points in the initial synthetic point cloud; and
sampling from the initial synthetic point cloud according to the probabilities of the probability mask to generate the synthetic LIDAR data such that a probability that the synthetic LIDAR data comprises a respective point from the initial synthetic point cloud is determined by a corresponding probability from the probability mask.

6. The autonomous vehicle control system of claim 1 , wherein the at least one of the one or more machine-learned models was tested by:
processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data.

7. The autonomous vehicle control system of claim 6 , wherein the at least one of the one or more machine-learned models was tested by:
generating additional synthetic LiDAR data based on motion controls output by a motion planning system based on the processing of the synthetic LiDAR data using the at least one of the one or more machine-learned models; and
processing the additional synthetic LiDAR data using the at least one of the one or more machine-learned models to test the performance of the at least one of the one or more machine-learned models in a different position in the test environment described by the additional synthetic LiDAR data.

8. The autonomous vehicle control system of claim 6 , wherein the at least one of the one or more machine-learned models was tested by:
inserting an additional mesh representation of a virtual object into the test environment before using the physics-based simulation engine to obtain the initial synthetic point cloud descriptive of the test environment to generate a specific test scenario associated with the additional mesh representation of the virtual object.

9. One or more non-transitory computer-readable media storing:
one or more machine-learned models, wherein at least one of the one or more machine-learned models was tested using synthetic light detection and ranging (LiDAR) data, wherein the synthetic LiDAR data was generated by:
using a physics-based simulation engine to obtain an initial synthetic point cloud,
generating, by a machine-learned model, a value corresponding to a probability that a real LiDAR point cloud would have an error at a point in the initial synthetic point cloud, and
generating, based on a determination of whether, based on the value, to include the error in the synthetic LIDAR data, the synthetic LIDAR data to contain the error at the point; and

instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations, the operations comprising:
obtaining LIDAR data descriptive of an environment of the autonomous vehicle;
determining, using the one or more machine-learned models and based on the LiDAR data, a motion plan for controlling the autonomous vehicle; and
controlling the autonomous vehicle according to the motion plan.

10. The one or more non-transitory computer-readable media of claim 9 , wherein using the machine-learned model to add realistic errors to the initial synthetic point cloud comprises:
sampling, based on the value, to determine whether to include the error at the point in the synthetic LIDAR data and.

11. The one or more non-transitory computer-readable media of claim 9 , wherein the probability corresponds to a dropout probability.

12. The one or more non-transitory computer-readable media of claim 11 , wherein generating, based on the sampling, the synthetic LIDAR data to contain the error at the point comprises omitting the point.

13. The one or more non-transitory computer-readable media of claim 9 , wherein using the machine-learned model to add realistic errors to the initial synthetic point cloud comprises:
generating, using the machine-learned model, a probability mask over the initial synthetic point cloud, the probability mask indicating respective probabilities that a real LiDAR point cloud would have an error at respective points in the initial synthetic point cloud; and
sampling from the initial synthetic point cloud according to the probabilities of the probability mask to generate the synthetic LIDAR data such that a probability that the synthetic LIDAR data comprises a respective point from the initial synthetic point cloud is determined by a corresponding probability from the probability mask.

14. The one or more non-transitory computer-readable media of claim 9 , wherein the at least one of the one or more machine-learned models was tested by:
processing the synthetic LiDAR data using the at least one of the one or more machine-learned models to test a performance of the at least one of the one or more machine-learned models in a test environment described by the synthetic LiDAR data.

15. The one or more non-transitory computer-readable media of claim 14 , wherein the at least one of the one or more machine-learned models was tested by:
generating additional synthetic LiDAR data based on motion controls output by a motion planning system based on the processing of the synthetic LiDAR data using the at least one of the one or more machine-learned models; and
processing the additional synthetic LiDAR data using the at least one of the one or more machine-learne

Assignees

• Aurora Operations Inc

Inventors

• Manivasagam Sivabalan
• Wang Shenlong
• Ma Wei-Chiu
• Wong Kelvin Ka Wing
• Zeng Wenyuan
• Urtasun Raquel

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T15/06

  Ray-tracing · CPC title