Landmark perception for localization in autonomous systems and applications

Patent metadata

Field	Value
Publication number	US-12482118-B2
Application number	US-202318171016-A
Country	US
Kind code	B2
Filing date	Feb 17, 2023
Priority date	Feb 17, 2023
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

In various examples, perception of landmark shapes may be used for localization in autonomous systems and applications. In some embodiments, a deep neural network (DNN) is used to generate (e.g., per-point) classifications of measured 3D points (e.g., classified LiDAR points), and a representation of the shape of one or more detected landmarks is regressed from the classifications. For each of one or more classes, the classification data may be thresholded to generate a binary mask and/or dilated to generate a densified representation, and the resulting (e.g., dilated, binary) mask may be clustered into connected components that are iteratively: fitted a shape (e.g., a polynomial or Bezier spline for lane lines, a circle for top-down representations of poles or traffic lights), weighted, and merged. As such, the resulting connected components and their fitted shapes may be used to represent detected landmarks and used for localization, navigation, and/or other uses.

First claim

What is claimed is:

1 . A method comprising:
clustering a representation of classification data into groups of connected pixels of a common class, the classification data representing one or more classifications of sensor data generated using one or more sensors of an ego-machine; generating one or more fitted shapes of one or more detected landmarks represented by the classification data based at least on performing one or more iterations of:
fitting one or more candidate fitted shapes to the groups of connected pixels of the common class; and
combining the groups of connected pixels of the common class based at least on the one or more candidate fitted shapes; and
executing one or more navigation or localization operations of the ego-machine based at least on the one or more fitted shapes of the one or more detected landmarks.

2 . The method of claim 1 , further comprising generating the classification data based on at least on processing, using one or more neural networks, a projected representation of LiDAR data accumulated over one or more spins of a LiDAR sensor.

3 . The method of claim 1 , further comprising generating the representation of the classification data based at least on thresholding and dilating the classification data.

4 . The method of claim 1 , wherein (i) the classification data represents one or more detected lane lines, and the generating the one or more fitted shapes comprises fitting one or more polynomials to the groups of connected pixels of the common class, or (ii) the classification data represents one or more detected poles or traffic lights, and the generating the one or more fitted shapes comprises fitting one or more circles to the groups of connected pixels of the common class.

5 . The method of claim 1 , wherein the generating the one or more fitted shapes comprises:
fitting at least one individual polynomial of one or more polynomials based at least on sampling a centroid pixel of a connected component of the groups of connected pixels of the common class; sampling a number of random pixels from the connected component; and fitting the at least one individual polynomial to the centroid pixel and the random pixels.

6 . The method of claim 1 , wherein the generating the one or more fitted shapes comprises determining not to fit a candidate shape to one or more of the groups of connected pixels of the common class having an area less than a threshold area.

7 . The method of claim 1 , wherein the combining the groups of connected pixels of the common class comprises absorbing a first connected component into a second connected component having a candidate fitted shape based at least on the candidate fitted shape of the second connected component intersecting the first connected component and having a higher weight than other candidate fitted shapes that intersect the first connected component.

8 . The method of claim 1 , wherein the combining the groups of connected pixels of the common class comprises weighting the one or more candidate fitted shapes based at least on a number and an area of a set of the groups of connected pixels of the common class intersected by individual candidate fitted shapes of the one or more candidate fitted shapes.

9 . The method of claim 1 , wherein the combining the groups of connected pixels of the common class comprises:
determining whether a candidate fitted shape of the one or more candidate fitted shapes intersects a connected component of the groups of connected pixels of the common class based at least on sampling a centroid pixel and a number of random pixels of the connected component; and evaluating whether the centroid pixel and the random pixels are within a threshold distance of the candidate fitted shape.

10 . The method of claim 1 , wherein the method is performed by at least one of:
a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system for performing digital twin operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for generating synthetic data; or a system implemented at least partially using cloud computing resources.

11 . A processor comprising:
one or more processing units to:
cluster a representation of classification data into one or more groups of connected pixels, the classification data representing one or more classifications of sensor data generated using one or more sensors of an ego-machine;
generate one or more fitted shapes of one or more detected landmarks represented in the classification data based at least on iteratively fitting one or more candidate fitted shapes to the one or more groups of connected pixels; and
execute one or more control operations of the ego-machine based at least on the one or more fitted shapes of the one or more detected landmarks.

12 . The processor of claim 11 , the one or more processing units further to generate the representation of the classification data based at least on thresholding and dilating the classification data.

13 . The processor of claim 11 , wherein (i) the classification data represents one or more detected lane lines, and the one or more processing units are further to generate the one or more fitted shapes based at least on fitting one or more polynomials to the one or more groups of connected pixels, or (ii) the classification data represents one or more detected poles or traffic lights, and the one or more processing units are further to generate the one or more fitted shapes based at least on fitting one or more circles to the one or more groups of connected pixels.

14 . The processor of claim 11 , the one or more processing units further to generate the one or more fitted shapes based at least on determining not to fit a candidate shape to at least one of the one or more groups of connected pixels having an area less than a threshold area.

15 . The processor of claim 11 , the one or more processing units further to generate the one or more fitted shapes based at least on absorbing a first group of connected pixels into a second group of connected pixels having a candidate fitted shape based at least on the candidate fitted shape of the second group of connected pixels intersecting the first group of connected pixels and having a higher weight than other candidate fitted shapes that intersect the first group of connected pixels.

16 . The processor of claim 11 , the one or more processing units further to generate the one or more fitted shapes based at least on weighting the one or more candidate fitted shapes based at least on a number and an area of a set of the one or more groups of connected pixels intersected by individual candidate fitted shapes of the one or more candidate fitted shapes.

17 . The processor of claim 11 , the one or more processing units further to generate the one or more fitted shapes based at least on determining whether a candidate fitted shape of the one or more candidate fitted shapes intersects a group

Assignees

• Nvidia Corp

Inventors

• Abbott Joshua Edward
• Akbarzadeh Amir
• Pehserl Joachim
• Ogden Samuel
• Wehr David
• Chen Ke

Classifications

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/30256

  Lane; Road marking · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G01S17/89 Primary

  for mapping or imaging · CPC title

• G06T2207/30184

  Infrastructure · CPC title