Lidar scene generation for training machine learning models

Patent metadata

Field	Value
Publication number	US-12249028-B2
Application number	US-202218070213-A
Country	US
Kind code	B2
Filing date	Nov 28, 2022
Priority date	Nov 28, 2022
Publication date	Mar 11, 2025
Grant date	Mar 11, 2025

Abstract

A lidar method and system for training and using a machine learning model are disclosed. An example method includes: obtaining a map of a background scene, the map including three-dimensional (3D) point cloud data; obtaining at least one point cloud representation of at least one foreground object, the at least one point cloud representation including 3D point cloud data, wherein one or more lidar sensors were used to generate the 3D point cloud data for the map and the at least one point cloud representation; generating a lidar scene by placing the at least one point cloud representation within the map; and training the machine learning model using the generated lidar scene. The model can be used to identify objects and control a vehicle.

First claim

What is claimed is:

1. A method of training a machine learning model, the method comprising:
obtaining a map of a background scene, the map comprising three-dimensional (3D) point cloud data;
obtaining, by retrieving from a library of point cloud images of a plurality of foreground objects, at least one point cloud representation of at least one foreground object, the at least one point cloud representation comprising 3D point cloud data,
wherein one or more light detection and ranging (lidar) sensors were used to generate the 3D point cloud data for the map and the at least one point cloud representation and
wherein the library comprises point cloud representations of the at least one foreground object in multiple orientations with respect to the one or more lidar sensors;

generating a lidar scene by placing the at least one point cloud representation within the map; and
training the machine learning model using the generated lidar scene.

2. The method of claim 1 , wherein the background scene comprises at least one of a residential area, a downtown area, a highway, a road, a sidewalk, a city, natural features, natural terrain, or any combination thereof.

3. The method of claim 1 , wherein obtaining the map comprises retrieving the map from a library comprising a plurality of maps for a plurality of background scenes.

4. The method of claim 1 , wherein obtaining the map of the background scene comprises:
performing one or more scans of the background scene using the one or more lidar sensors;
identifying a dynamic object in the one or more scans; and
removing the dynamic object from the one or more scans.

5. The method of claim 1 , wherein the at least one foreground object comprises a car, a bus, a motorcycle, a bicycle, a vehicle, a pedestrian, a person, an animal, or any combination thereof.

6. The method of claim 1 , further comprising:
scanning, by the one or more lidar sensors, the at least one foreground object while the at least one foreground object is on a turntable, thereby obtaining the point cloud representations of the at least one foreground object in multiple orientations with respect to the one or more lidar sensors; and
storing the point cloud representations of the at least one foreground object in the library of point cloud images.

7. The method of claim 1 , wherein the lidar scene comprises at least one annotation identifying the at least one foreground object.

8. The method of claim 1 , wherein generating the lidar scene comprises selecting a lidar sensor position within the map.

9. The method of claim 1 , further comprising:
generating a second lidar scene by placing the at least one point cloud representation within a second map of a second background scene, the second map comprising 3D point cloud data; and
training the machine learning model using the generated second lidar scene.

10. A system for training a machine learning model, the system comprising:
one or more computer processors programmed to perform operations comprising:
obtaining a map of a background scene, the map comprising three-dimensional (3D) point cloud data;
obtaining, by retrieving from a library of point cloud images of a plurality of foreground objects, at least one point cloud representation of at least one foreground object, the at least one point cloud representation comprising 3D point cloud data,
wherein one or more light detection and ranging (lidar) sensors were used to generate the 3D point cloud data for the map and the at least one point cloud representation and
wherein the library comprises point cloud representations of the at least one foreground object in multiple orientations with respect to the one or more lidar sensors;

generating a lidar scene by placing the at least one point cloud representation within the map; and
training the machine learning model using the generated lidar scene.

11. The system of claim 10 , wherein the background scene comprises at least one of a residential area, a downtown area, a highway, a road, a sidewalk, a city, natural features, natural terrain, or any combination thereof.

12. The system of claim 10 , wherein obtaining the map comprises retrieving the map from a library comprising a plurality of maps for a plurality of background scenes.

13. The system of claim 10 , wherein obtaining the map of the background scene comprises:
performing one or more scans of the background scene using the one or more lidar sensors;
identifying a dynamic object in the one or more scans; and
removing the dynamic object from the one or more scans.

14. The system of claim 10 , wherein the at least one foreground object comprises a car, a bus, a motorcycle, a bicycle, a vehicle, a pedestrian, a person, an animal, or any combination thereof.

15. The system of claim 10 , wherein the one or more computer processors include at least one processor programmed to perform operations comprising:
scanning, by the one or more lidar sensors, the at least one foreground object while the at least one foreground object is on a turntable, thereby obtaining the point cloud representations of the at least one foreground object in multiple orientations with respect to the one or more lidar sensors; and
storing the point cloud representations of the at least one foreground object in the library of point cloud images.

16. A method of controlling a vehicle, comprising:
collecting data using a light detection and ranging (lidar) sensor on the vehicle while the vehicle is being operated;
providing the data to a machine learning model,
wherein the machine learning model has been trained using a generated lidar scene, and
wherein the generated lidar scene comprises a lidar point cloud representation of at least one foreground object placed within a map of a background scene comprising lidar point cloud data,
wherein the lidar point cloud representation of the at least one foreground object is obtained from a library of point cloud images of a plurality of foreground objects, the library comprising point cloud representations of the at least one foreground object in multiple orientations with respect to the one or more lidar sensors,

receiving from the machine learning model an identification of an object in a vicinity of the vehicle; and
controlling the vehicle based on the identification of the object in the vicinity of the vehicle.

Assignees

• Velodyne Lidar Usa Inc

Inventors

• Ranganath Deepak Guntur
• Jain Vishal

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T2210/56

  Particle system, point based geometry or rendering · CPC title

• G01S17/89

  for mapping or imaging · CPC title