Deep neural network for detecting obstacle instances using radar sensors in autonomous machine applications

Patent metadata

Field	Value
Publication number	US-12399253-B2
Application number	US-202318493452-A
Country	US
Kind code	B2
Filing date	Oct 24, 2023
Priority date	Nov 21, 2019
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

In various examples, a deep neural network(s) (e.g., a convolutional neural network) may be trained to detect moving and stationary obstacles from RADAR data of a three dimensional (3D) space, in both highway and urban scenarios. RADAR detections may be accumulated, ego-motion-compensated, orthographically projected, and fed into a neural network(s). The neural network(s) may include a common trunk with a feature extractor and several heads that predict different outputs such as a class confidence head that predicts a confidence map and an instance regression head that predicts object instance data for detected objects. The outputs may be decoded, filtered, and/or clustered to form bounding shapes identifying the location, size, and/or orientation of detected object instances. The detected object instances may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

First claim

What is claimed is:

1. One or more processors of a machine in an environment, the one or more processors to:
generate one or more bounding shapes corresponding to one or more detected objects in the environment, the one or more bounding shapes generated based at least on class confidence data and instance regression data, the class confidence data comprising a plurality of classification channels, and the instance regression data comprising, a plurality of regression channels for each classification channel of the plurality of classification channels, a plurality of regression channels individually corresponding to one or more regressed candidate bounding shapes one or more bounding shapes corresponding to one or more detected objects in the environment; and
cause performance of one or more control operations associated with the machine based at least on a representation of the one or more bounding shapes.

2. The one or more processors of claim 1 , wherein the one or more processors are further to generate the one or more bounding shapes for a first class of the one or more detected objects using a confidence map from one of the plurality of classification channels for the first class and a corresponding subset of the plurality of regression channels for the first class.

3. The one or more processors of claim 1 , wherein the one or more processors are further to generate the one or more regressed candidate bounding shapes for each class of a plurality of classes based at least on decoding the instance regression data from a subset of the plurality of regression channels corresponding to each class.

4. The one or more processors of claim 1 , wherein the one or more processors are further to decode the one or more regressed candidate bounding shapes for a first class using a subset of the plurality of regression channels corresponding to the first class, and associate each of the one or more regressed candidate bounding shapes with a corresponding confidence value associated with one or more corresponding pixels from a corresponding one of the plurality of classification channels for the first class.

5. The one or more processors of claim 1 , wherein the one or more processors are further to decode the one or more regressed candidate bounding shapes for a first class using a subset of the plurality of regression channels corresponding to the first class, and assign to each regressed candidate bounding shape of the one or more regressed candidate bounding shapes a corresponding confidence value of a representative pixel of the regressed candidate bounding shape retrieved from a corresponding one of the plurality of classification channels for the first class.

6. The one or more processors of claim 1 , wherein the one or more processors are further to decode the one or more regressed candidate bounding shapes for a first class using a subset of the plurality of regression channels corresponding to the first class, and assign to each regressed candidate bounding shape of the one or more regressed candidate bounding shapes a corresponding composite confidence value derived from pixels of the regressed candidate bounding shape from a corresponding one of the plurality of classification channels for the first class.

7. The one or more processors of claim 1 , wherein the one or more processors are further to decode the one or more regressed candidate bounding shapes for a first class using a subset of the plurality of regression channels corresponding to the first class, and filter out at least one of the one or more regressed candidate bounding shapes based at least one or more corresponding confidence values of one or more corresponding pixels from a corresponding one of the plurality of classification channels for the first class.

8. The one or more processors of claim 1 , wherein the one or more regressed candidate bounding shapes corresponding to the plurality of regression channels for at least one individual classification channel of the plurality of classification channels are associated with a corresponding class of the at least one individual classification channel.

9. The one or more processors of claim 1 , wherein the one or more processors are comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system for performing remote operations;
a system for performing real-time streaming;
a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content;
a system implemented using a robot;
a system for generating synthetic data;
a system for generating synthetic data using AI; or
a system implemented at least partially using cloud computing resources.

10. A system comprising one or more processors of a machine in an environment, the one or more processors to:
generate one or more bounding shapes corresponding to one or more detected objects in the environment based at least on a class confidence tensor comprising a plurality of classification channels and an instance regression tensor comprising, for each classification channel of the plurality of classification channels, a plurality of regression channels representing regressed object instance data; and
cause performance of one or more control operations associated with the machine based at least on a representation of the one or more bounding shapes.

11. The system of claim 10 , wherein the one or more processors are further to generate the one or more bounding shapes for a first class of the one or more detected objects using a confidence map from one of the plurality of classification channels for the first class and a corresponding subset of the plurality of regression channels for the first class.

12. The system of claim 10 , wherein the one or more processors are further to generate one or more candidate bounding shapes for each class of a plurality of classes based at least on decoding the regressed object instance data from a subset of the plurality of regression channels predicted for each class.

13. The system of claim 10 , wherein the one or more processors are further to decode one or more candidate bounding shapes for a first class using the regressed object instance data from a subset of the plurality of regression channels predicted for the first class, and associate each of one or more candidate bounding shapes with a corresponding confidence value associated with one or more corresponding pixels from a corresponding one of the plurality of classification channels for the first class.

14. The system of claim 10 , wherein the one or more processors are further to decode one or more candidate bounding shapes for a first class using the regressed object instance data from a subset of the plurality of regression channels predicted for the first class, and assign to each candidate bounding shape of the one or more candidate bounding shapes a corresponding confidence value of a representative pixel of the candidate bounding shape retrieved from a corresponding one of the plurality of classification channels for the first class.

15. The system of claim 10 , wherein the one or more processors are further to decode one or more candidate bounding shapes for a first class using the regressed object instance data from a subset of the plurality of regression channels predicted for the first class, and assign to each candidate bounding shape of the one or more candidate bounding shapes a corresponding composite confidence value derived from pixels of the candidate bound

Assignees

• Nvidia Corp

Inventors

• Popov Alexander
• Smolyanskiy Nikolai
• Oldja Ryan
• Murray Shane
• Wekel Tilman
• Nister David
• Pehserl Joachim
• Bhargava Ruchi
• Oh Sangmin

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06V10/82

  using neural networks · CPC title