Transferring synthetic lidar system data to real world domain for autonomous vehicle training applications

Patent metadata

Field	Value
Publication number	US-11734935-B2
Application number	US-202117314462-A
Country	US
Kind code	B2
Filing date	May 7, 2021
Priority date	Apr 10, 2019
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

Methods and systems are disclosed for correlating synthetic LiDAR data to a real-world domain for use in training an model for use by autonomous vehicle when operating in an environment. To do this, the system will obtain a data set of synthetic LiDAR data, along with images of a real-world environment. The system will transfer the synthetic LiDAR data to a two-dimensional representation, use the two-dimensional representation and the images to train a model that a vehicle can use to operate in a real-world environment.

First claim

The invention claimed is:

1. A method of correlating or transforming synthetic LiDAR data to a real-world domain for training an autonomous vehicle, the method comprising, by a processing device:
obtaining a data set of synthetic LiDAR data;
obtaining a plurality of images comprising representations of real-world objects;
transferring the synthetic LiDAR data to a two-dimensional representation by:
generating a plurality of two-dimensional front-view images by converting each 3D point of the synthetic LiDAR data into a pixel, the two-dimensional front-view images comprising representations of simulated objects;
obtaining a mapping that relates one or more of the simulated objects to one or more of the real-world objects represented in a plurality of images; and
annotating the plurality of two-dimensional front-view images by adding bounding boxes around the simulated objects in accordance with the mapping;

using the two-dimensional representation to train a model of the real-world environment; and
using the trained model of the real-world environment to train an autonomous vehicle.

2. The method of claim 1 , wherein using the two-dimensional representation to train the model of the real-world environment comprises inputting two-dimensional front view images generated from the synthetic LiDAR data into a convolutional neural network.

3. The method of claim 2 , further comprising generating the two-dimensional front view images from the synthetic LiDAR data by extracting front view depth images from the synthetic LiDAR data, in which substantially each pixel of each front view depth image is associated with a corresponding depth measurement.

4. The method of claim 1 , wherein obtaining the data set of synthetic LiDAR data comprises using a simulation application to generate the data set of synthetic LiDAR data by operating a virtual vehicle in a simulated environment of the simulation application.

5. The method of claim 4 , wherein using the simulation application to generate the data set of synthetic LiDAR data comprises, as the virtual vehicle is operated in the simulated environment:
accessing a plurality of depth images generated by the simulation;
extracting samples from a plurality of image frames generated by the simulation; and
correlating the extracted samples to the depth images to approximate each extracted sample to a pixel in one or more of the depth images.

6. The method of claim 1 , wherein transferring the synthetic LiDAR data to the two-dimensional representation comprises transferring the synthetic LiDAR data to a birds-eye view representation.

7. The method of claim 1 , further comprising generating ground truth annotations in a plurality of frames of synthetic LiDAR data to identify one or more objects in the synthetic LiDAR data.

8. The method of claim 1 , wherein:
the method further comprises generating ground truth annotations in a plurality of frames of the data set of synthetic LiDAR data to identify one or more objects in the synthetic LiDAR data; and
wherein at least some of the plurality of two-dimensional front-view images comprise one or more of the ground truth annotations that identify one or more objects.

9. The method of claim 1 wherein using the two-dimensional representation to train the model of the real-world environment comprises using the plurality of images comprising representations of the real-world objects, the two-dimensional front-view images from the LiDAR data, and a result of the mapping to train the model of the real-world environment.

10. A system for correlating synthetic LiDAR data to a real-world domain for an autonomous vehicle, the system comprising:
a processor;
a non-transitory memory that stores a data set of synthetic LiDAR data; and
a non-transitory memory containing programming instructions that are configured to cause the processor to:
cause the system to receive a plurality of images captured from a real-world environment;
access the data set of synthetic LiDAR data and extract synthetic LiDAR data from the data set,
transfer the extracted synthetic LiDAR data to a two-dimensional representation by:
generating a plurality of two-dimensional front-view images by converting each 3D point of the synthetic LiDAR data into a pixel, the two-dimensional front-view images comprising representations of simulated objects; and
obtaining a mapping that relates one or more of the simulated objects to one or more of the real-world objects represented in the plurality of images; and
annotating the plurality of two-dimensional front-view images by adding bounding boxes around the simulated objects in accordance with the mapping;

use the two-dimensional representation to train a model of the real-world environment; and
use the trained model of the real-world environment to train an autonomous vehicle.

11. The system of claim 10 , wherein the instructions to use the two-dimensional representation to train the model of the real-world environment comprise instructions to input two-dimensional front view images generated from the synthetic LiDAR data into a convolutional neural network.

12. The system of claim 11 , further comprising instructions to generate the two-dimensional front view images from the synthetic LiDAR data by extracting front view depth images from the synthetic LiDAR data, in which substantially each pixel of each front view depth image is associated with a corresponding depth measurement.

13. The system of claim 10 , further comprising additional programming instructions that are configured to cause a processor to implement a simulation application that generates the data set of synthetic LiDAR data by operating a virtual vehicle in a simulated environment of the simulation application.

14. The system of claim 13 , wherein the programming instructions to implement the simulation application that generates the data set of synthetic LiDAR data comprise instructions to, as the virtual vehicle is operated in the simulated environment:
access a plurality of depth images generated by the simulation;
extract samples from a plurality of image frames generated by the simulation; and
correlate the extracted samples to the depth images to approximate each extracted sample to a pixel in one or more of the depth images.

15. The system of claim 10 , wherein the programming instructions to transfer the synthetic LiDAR data to the two-dimensional representation comprises instructions to transfer the synthetic LiDAR data to a birds-eye view representation.

16. The system of claim 10 , further comprising additional programming instructions that are configured to cause the processor to generate ground truth annotations in a plurality of frames of synthetic LiDAR data to identify one or more objects in the synthetic LiDAR data.

17. The system of claim 10 :
further comprising additional programming instructions that are configured to cause the processor to generate ground truth annotations in a plurality of frames of the data set of synthetic LiDAR data to identify one or more objects in the synthetic LiDAR data; and
wherein at least some of the plurality of two-dimensional front-view images comprise one or more of the ground truth annotations that identify one or more objects.

18. The system of claim 10 , wherein the instructions to use the two-dimensional representation to train the model of the real-world environment comprise instructions to use the plurality of images captured from the real-world environment, the two-dimensional front-view images from the LiDAR data, and a r

Assignees

• Argo Ai Llc

Inventors

• Chen Kevin
• Hays James
• Yumer Ersin

Classifications

• G06T3/067

  Reshaping or unfolding three-dimensional [3D] tree structures onto two-dimensional [2D] planes · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G05D1/0221

  involving a learning process · CPC title

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G01S17/90

  using synthetic aperture techniques · CPC title