Systems and methods for generating synthetic sensor data via machine learning

Patent metadata

Field	Value
Publication number	US-11544167-B2
Application number	US-202016826990-A
Country	US
Kind code	B2
Filing date	Mar 23, 2020
Priority date	Mar 23, 2019
Publication date	Jan 3, 2023
Grant date	Jan 3, 2023

Abstract

The present disclosure provides systems and methods that combine physics-based systems with machine learning to generate synthetic LiDAR data that accurately mimics a real-world LiDAR sensor system. In particular, aspects of the present disclosure combine physics-based rendering with machine-learned models such as deep neural networks to simulate both the geometry and intensity of the LiDAR sensor. As one example, a physics-based ray casting approach can be used on a three-dimensional map of an environment to generate an initial three-dimensional point cloud that mimics LiDAR data. According to an aspect of the present disclosure, a machine-learned model can predict one or more dropout probabilities for one or more of the points in the initial three-dimensional point cloud, thereby generating an adjusted three-dimensional point cloud which more realistically simulates real-world LiDAR data.

First claim

What is claimed is:

1. A computer-implemented method to generate synthetic light detection and ranging (LiDAR) data, the method comprising:
obtaining, by a computing system comprising one or more computing devices, a three-dimensional map of an environment;
determining, by the computing system, a trajectory that describes a series of locations of a virtual object relative to the environment over time;
performing, by the computing system, ray casting on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points,
wherein performing, by the computing system, the ray casting to generate the initial three-dimensional point cloud comprises determining, by the computing system for each of a plurality of rays, a ray casting location and a ray casting direction based at least in part on the trajectory;

processing, by the computing system using a machine-learned model, the initial three-dimensional point cloud to predict a respective dropout probability for one or more of the plurality of points; and
generating, by the computing system, an adjusted three-dimensional point cloud from the initial three-dimensional point cloud based at least in part on the respective dropout probabilities predicted by the machine-learned model for the one or more of the plurality of points of the initial three-dimensional point cloud.

2. The computer-implemented method of claim 1 , wherein generating, by the computing system, an adjusted three-dimensional point cloud from the initial three-dimensional point cloud based at least in part on the respective dropout probabilities predicted by the machine-learned model for the one or more of the plurality of points of the initial three-dimensional point cloud comprises removing, by the computing system, each of one of the one or more of the plurality of points with probability equal to its respective dropout probability.

3. The computer-implemented method of claim 1 , wherein processing, by the computing system using the machine-learned model, the initial three-dimensional point cloud to predict the respective dropout probability for one or more of the plurality of points comprises:
transforming, by the computing system, the initial three-dimensional point cloud into a two-dimensional polar image grid; and
processing, by the computing system using the machine-learned model, the two-dimensional polar image grid to generate a two-dimensional ray dropout probability map.

4. The computer-implemented method of claim 1 , wherein performing, by the computing system, the ray casting to generate the initial three-dimensional point cloud comprises:
identifying, by the computing system for each of the plurality of rays, a closest surface element in the three-dimensional map to the ray casting location and along the ray casting direction; and
generating, by the computing system for each of the plurality of rays, one of the plurality of points with a respective depth based at least in part on a distance from the ray casting location to the closest surface element.

5. The computer-implemented method of claim 1 , further comprising feeding, by the computing system, the adjusted three-dimensional point cloud as LiDAR data input to an autonomy computing system of an autonomous vehicle to test a performance of the autonomy computing system of the autonomous vehicle in the environment.

6. The computer-implemented method of claim 1 , wherein the machine-learned model comprises a U-Net neural network.

7. The computer-implemented method of claim 1 , wherein obtaining, by the computing system, the three-dimensional map of the environment comprises generating, by the computing system, the three-dimensional map, and generating, by the computing system, the three-dimensional map comprises:
obtaining, by the computing system, a plurality of sets of real-world LiDAR data physically collected by one or more LiDAR systems in the environment;
removing, by the computing system, one or more moving objects from the plurality of sets of real-world LiDAR data;
associating, by the computing system, the plurality of sets of real-world LiDAR data to a common coordinate system to generate an aggregate LiDAR point cloud; and
converting, by the computing system, the aggregate LiDAR point cloud to a surface element-based three-dimensional mesh.

8. The computer-implemented method of claim 1 , wherein the machine-learned model has been trained using an objective function that computes a pixel-wise loss that compares a predicted dropout probability map with a ground truth dropout mask.

9. The computer-implemented method of claim 1 , further comprising:
inserting, by the computing system, one or more dynamic virtual objects into the three-dimensional map of the environment;
wherein performing, by the computing system, ray casting on the three-dimensional map comprises performing, by the computing system, ray casting on the three-dimensional map including the one or more dynamic virtual objects.

10. A computing system, comprising:
one or more processors;
a machine-learned model configured to predict dropout probabilities for LiDAR data; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a ground truth three-dimensional point cloud collected by a physical LiDAR system as the physical LiDAR system travelled along a trajectory through an environment;
generating a ground truth dropout mask for the ground truth three-dimensional point cloud;
obtaining a three-dimensional map of the environment;
performing ray casting on the three-dimensional map according to the trajectory to generate an initial three-dimensional point cloud that comprises a plurality of points;
processing, using the machine-learned model, the initial three-dimensional point cloud to generate a dropout probability map that provides a respective dropout probability for one or more of the plurality of points of the initial three-dimensional point cloud;
evaluating an objective function that compares the dropout probability map generated by the machine-learned model to the ground truth dropout mask; and
modifying one or more values of one or more parameters of the machine-learned model based at least in part on the objective function.

11. The computing system of claim 10 , wherein each of the ground truth dropout mask and the dropout probability map comprises a two-dimensional polar image grid.

12. The computing system of claim 11 , wherein evaluating the objective function comprises determining a pixel-wise binary cross entropy between the ground truth dropout mask and the dropout probability map.

13. The computing system of claim 10 , wherein modifying the one or more values of the one or more parameters of the machine-learned model based at least in part on the objective function comprises backpropagating the objective function through the machine-learned model.

14. The computing system of any of claim 10 , wherein the machine-learned model comprises U-Net neural network.

15. One or more non-transitory computer-readable media that collectively store instructions that, when executed by a computing system comprising one or more computing devices, cause the computing system to generate three-dimensional representations of objects by performing operations, the operations comprising:
obtaining, by the computing system, one or more sets of real-world LiDAR data physically collected by one or more LiDAR sys

Assignees

• Uatc Llc

Inventors

• Manivasagam Sivabalan
• Wang Shenlong
• Ma Wei-Chiu
• Wong Kelvin Ka Wing
• Zeng Wenyuan
• Urtasun Raquel

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G01S17/931

  of land vehicles · CPC title

• G06T17/05

  Geographic models · CPC title

• G06T15/06

  Ray-tracing · CPC title