Analysis of point cloud data using depth maps

Patent metadata

Field	Value
Publication number	US-11908203-B2
Application number	US-202217718721-A
Country	US
Kind code	B2
Filing date	Apr 12, 2022
Priority date	Feb 27, 2018
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

LiDAR (light detection and ranging) and RADAR (radio detection and ranging) systems are commonly used to generate point cloud data for 3D space around vehicles, for such functions as localization, mapping, and tracking. Improved techniques for processing the point cloud data that has been collected are provided. The improved techniques include mapping one or more point cloud data points into a depth map, the one or more point cloud data points being generated using one or more sensors; determining one or more mapped point cloud data points within a bounded area of the depth map, and detecting, using one or more processing units and for an environment surrounding a machine corresponding to the one or more sensors, a location of one or more entities based on the one or more mapped point cloud data points.

First claim

What is claimed is:

1. A method comprising:
mapping one or more motion-compensated point cloud data points into a depth map, the one or more point cloud data points being generated using one or more sensors;
determining one or more mapped point cloud data points within a bounded area of the depth map; and
detecting, using one or more processing units and for an environment surrounding a machine corresponding to the one or more sensors, a location of one or more entities based on the one or more mapped point cloud data points.

2. The method of claim 1 , wherein the one or more entities comprises at least one of: a location of the machine, an object in the environment, a traversable freespace in the environment, or a landmark in the environment.

3. The method of claim 1 , wherein said mapping comprises transforming the one or more point cloud data points into a polar depth map that represents one or more distances between the one or more point cloud data points and the one or more sensors.

4. The method as recited in claim 3 , further comprising:
determining plane information for the data points from the polar depth map; and
storing the plane information and the polar depth map using a two dimensional (2D) depth map, wherein said storing includes transforming the data points in the polar depth map into Euclidean coordinates using nearest neighbor analysis.

5. The method as recited in claim 4 , wherein said storing further includes computing a depth value for the one or more of the point cloud data points by intersecting a nominal ray with a plane corresponding to the one or more point cloud data points, and storing the depth values in the 2D depth map.

6. The method as recited in claim 4 , wherein said storing further includes using a texture map and analyzing the texture map using processing constructs of the one or more processing units.

7. The method as recited in claim 4 , wherein said determining plane information utilizes a best fit model over a bounded window of the point cloud data points.

8. The method as recited in claim 7 , wherein the bounded window defines a block of the point cloud data points, on which a nearest neighbor analysis is performed.

9. The method as recited in claim 7 , wherein the bounded window utilizes a fixed grid of the point cloud data points.

10. The method as recited in claim 1 , further comprising:
using the one or more sensors to generate, at a periodic interval, multiple slices of the point cloud data points.

11. The method as recited in claim 10 , further comprising:
estimating one or more trajectory equations using nearest neighbor analysis and the multiple slices of the point cloud data points.

12. The method as recited in claim 11 , further comprising:
calculating a movement of the object utilizing the one or more trajectory equations, wherein said calculating includes identifying a future position of the object using at least one of the one or more trajectory equations that corresponds to the object.

13. The method as recited in claim 11 , further comprising:
calculating a trajectory of the machine utilizing the one or more trajectory equations, wherein said calculating includes identifying motion vectors for the machine and surrounding objects from the multiple slices of the point cloud data points and aggregating identified motion vectors.

14. The method as recited in claim 11 , further comprising:
calculating a trajectory of the machine by producing transformations among the slices of the point cloud data points and inversing the transformations.

15. The method as recited in claim 10 , further comprising:
storing each of the slices as a portion of the depth map; and
using, for each of the slices, a first angle parameter for a horizontal dimension, and a second angle parameter for a vertical dimension.

16. The method as recited in claim 10 , further comprising:
estimating one or more trajectory equations by matching some of the mapped point cloud data points in the slices that correspond to a same object and calculating a distance between the some of the matched data points.

17. The method as recited in claim 10 , further comprising:
varying a size of the bounded area of the depth map based an angular separation between the slices.

18. The method as recited in claim 10 , further comprising:
calculating a movement of the object and a trajectory of the machine and determine an action that would prevent the object from coming into contact with the machine based on the movement of the object or the trajectory of the machine.

19. The method as recited in claim 1 , wherein the one or more sensors are part of a LiDAR system, a RADAR system or a camera system.

20. The method as recited in claim 1 , wherein said mapping comprises transforming the one or more point cloud data points to a two dimensional (2D) depth map.

21. The method as recited in claim 1 , further comprising:
determining a neighbor data point that is nearest to each of the one or more mapped point cloud data points within the bounded area.

22. The method as recited in claim 1 , wherein said mapping includes recording a single depth value for a set of the mapped point cloud data points that represent a region of the depth map.

23. The method as recited in claim 22 , wherein the single depth value is an average depth value of the region or the minimum depth value in the region.

24. A system comprising:
one or more processing units to perform one or more operations, the one or more operations including:
obtaining one or more data points of a point cloud, the one or more point cloud data points being generated using one or more sensors;
generating, for at least one data point of the one or more point cloud data points, at least one motion-compensated data point by computing a motion compensation for the at least one data point based at least on a motion of at least one sensor of the one or more sensors;
mapping the one or more point cloud data points, including the at least one motion-compensated data point, into a depth map;
determining one or more mapped point cloud data points within a bounded area of the depth map; and
detecting, using one or more processing units and for an environment surrounding a machine corresponding to the one or more sensors, a location of one or more entities in the environment based on the one or more mapped point cloud data points.

25. The system as recited in claim 24 , further comprising using the one or more sensors to generate multiple slices of the one or more point cloud data points.

26. A processor comprising:
one or more processing units to associate one or more motion-compensated data points of a point cloud to a bounded area of a depth map, and to detect a location of one or more entities in an environment using the associated one or more motion-compensated data points, wherein the one or more motion-compensated data points are obtained from one or more sensors observing the environment.

Assignees

• Nvidia Corp

Inventors

• Kulkarni Ishwar
• Eden Ibrahim
• Kroepfl Michael
• Nister David

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06V20/58 Primary

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G06T7/30

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title

• G06T7/521

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title