Systems and methods for generating synthetic sensor data via machine learning

Patent metadata

Field	Value
Publication number	US-11797407-B2
Application number	US-202217727085-A
Country	US
Kind code	B2
Filing date	Apr 22, 2022
Priority date	Mar 23, 2019
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining a three-dimensional map of an environment;
determining a trajectory that describes a series of locations of a virtual object relative to the environment over time;
performing a ray casting simulation on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points descriptive of at least a portion of the environment, wherein performing the ray casting simulation on the three-dimensional map according to the trajectory comprises determining, for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory;
processing, using a machine-learned model, the initial three-dimensional point cloud to predict a respective probability of an error for a respective point of the plurality of points of the initial three-dimensional point cloud; and
generating an adjusted three-dimensional point cloud from the initial three-dimensional point cloud by modifying geometry of the initial three-dimensional point cloud to include the error based at least in part on the respective probability of the error.

2. The computer-implemented method of claim 1 , wherein the respective probability of the error comprises a dropout probability.

3. The computer-implemented method of claim 2 , wherein generating the adjusted three-dimensional point cloud from the initial three-dimensional point cloud comprises removing the respective point based on the respective probability of the error for the respective point.

4. The computer-implemented method of claim 2 , wherein processing, using the machine-learned model, the initial three-dimensional point cloud to predict the respective probability of the error for the respective point of the plurality of points of the initial three-dimensional point cloud comprises:
transforming the initial three-dimensional point cloud into a two-dimensional polar image grid; and
processing, using the machine-learned model, the two-dimensional polar image grid to generate a two-dimensional ray dropout probability map.

5. The computer-implemented method of claim 1 , wherein performing the ray casting simulation on the three-dimensional map according to the trajectory to generate the initial three-dimensional point cloud comprises:
identifying, for each of the plurality of rays, a closest surface element in the three-dimensional map to the ray casting location and along the ray casting direction; and
generating, for each of the plurality of rays, one of the plurality of points with a respective depth based at least in part on a distance from the ray casting location to the closest surface element.

6. The computer-implemented method of claim 1 , further comprising feeding the adjusted three-dimensional point cloud as LiDAR data input to an autonomy computing system of an autonomous vehicle to test a performance of the autonomy computing system of the autonomous vehicle in the environment.

7. The computer-implemented method of claim 1 , wherein obtaining the three-dimensional map of the environment comprises generating the three-dimensional map, and wherein generating the three-dimensional map comprises:
obtaining a plurality of sets of real-world LiDAR data physically collected by one or more LiDAR systems in the environment;
removing one or more moving objects from the plurality of sets of real-world LiDAR data;
associating the plurality of sets of real-world LiDAR data to a common coordinate system to generate an aggregate LiDAR point cloud; and
converting the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh.

8. The computer-implemented method of claim 1 , wherein the machine-learned model has been trained using an objective function that comprises a pixel-wise loss that compares a predicted dropout probability map with a ground truth dropout mask.

9. The computer-implemented method of claim 1 , further comprising:
inserting one or more dynamic virtual objects into the three-dimensional map of the environment;
wherein performing ray casting on the three-dimensional map comprises performing ray casting on the three-dimensional map including the one or more dynamic virtual objects.

10. A computing system, comprising:
one or more processors;
a machine-learned model configured to predict probabilities of error for LiDAR data; and
one or more non-transitory computer-readable media that store instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining a ground truth three-dimensional point cloud collected by a physical LiDAR system as the physical LiDAR system travelled along a trajectory through an environment;
generating a ground truth mask for the ground truth three-dimensional point cloud;
obtaining a three-dimensional map of the environment;
performing ray casting on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points;
processing, using the machine-learned model, the initial three-dimensional point cloud to generate a probability of error map that provides a respective probability of error for one or more of the plurality of points of the initial three-dimensional point cloud;
evaluating an objective function that compares the probability of error map generated by the machine-learned model to the ground truth mask; and
modifying one or more values of one or more parameters of the machine-learned model based at least in part on the objective function.

11. The computing system of claim 10 , wherein the ground truth mask comprises a ground truth dropout mask and the probability of error map comprises a dropout probability map.

12. The computing system of claim 11 , wherein each of the ground truth dropout mask and the dropout probability map comprises a two-dimensional polar image grid.

13. The computing system of claim 11 , wherein evaluating the objective function comprises determining a pixel-wise binary cross entropy between the ground truth dropout mask and the dropout probability map.

14. The computing system of claim 10 , wherein modifying the one or more values of the one or more parameters of the machine-learned model based at least in part on the objective function comprises backpropagating the objective function through the machine-learned model.

15. One or more non-transitory computer-readable media that store instructions that are executable by a computing system comprising one or more computing devices to cause the computing system to generate three-dimensional representations of objects by performing operations, the operations comprising:
obtaining one or more sets of real-world LiDAR data physically collected by one or more LiDAR systems in a real-world environment, the one or more sets of real-world LiDAR data respectively comprising one or more three-dimensional point clouds;
defining a three-dimensional bounding box for an object included in the real-world environment;
identifying points from the one or more three-dimensional point clouds that are included within the three-dimensional bounding box to generate a set of accumulated points; and
generating a three-dimensional model of the object based at least in part on the set of accumulated points by:
mirroring the set of accumulated points along at least one axis of the three-dimensional bounding box to generate a set of mirrored points;
concatenating the set of mirrored points with the set of accumulated

Assignees

• Uatc Llc

Inventors

• Manivasagam Sivabalan
• Wang Shenlong
• Ma Wei-Chiu
• Wong Kelvin Ka Wing
• Zeng Wenyuan
• Urtasun Raquel

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F11/263 Primary

  Generation of test inputs, e.g. test vectors, patterns or sequences {; with adaptation of the tested hardware for testability with external testers} · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title