Deep learning based operational domain verification using camera-based inputs for autonomous systems and applications

Patent metadata

Field	Value
Publication number	US-12555385-B2
Application number	US-202117540654-A
Country	US
Kind code	B2
Filing date	Dec 2, 2021
Priority date	Dec 2, 2021
Publication date	Feb 17, 2026
Grant date	Feb 17, 2026

Abstract

In various examples, methods and systems are provided for determining, using a machine learning model, one or more of the following operational domain conditions related to an autonomous and/or semi-autonomous machine: amount of camera blindness, blindness classification, illumination level, path surface condition, visibility distance, scene type classification, and distance to a scene. Once one or more of these conditions are determined, an operational level of the machine may be determined, and the machine may be controlled according to the operational level.

First claim

What is claimed is:

1 . A processor comprising:
one or more circuits to:
compute, using one or more neural networks and based at least on image data generated using one or more image sensors, first data indicating a visibility distance associated with the one or more image sensors is reduced, the visibility distance associated with the image data;
determine, based at least on the image data, a confidence associated with a detected object;
based at least on the first data indicating that the visibility distance is reduced, determine, based at least on a distance to the detected object being further than the visibility distance, to reduce the confidence associated with the detected object;
determine, based at least in part on the confidence as reduced, one or more operations for an ego-machine; and
control the ego-machine according to the one or more operations.

2 . The processor of claim 1 , wherein the first data is further representative of at least one of:
a blindness classification associated with the one or more sensors; an illumination level associated with the one or more sensors; or a blindness level associated with the one or more sensors.

3 . The processor of claim 1 , wherein the one or more circuits are further to:
generate intermediate data using the one or more neural networks; and compute a subset of the first data based at least on the intermediate data and at least one head of the one or more neural networks, wherein the subset of the first data is representative of at least one of:
a sensor blindness level associated with the one or more sensors;
a blindness classification associated with the one or more sensors;
an illumination level associated with the one or more sensors; or
the visibility distance associated with the one or more sensors.

4 . The processor of claim 3 , wherein the at least one head includes one or more down-sampling layers followed by one or more up-sampling layers.

5 . The processor of claim 3 , wherein the at least one head includes at least one fully connected layer.

6 . The processor of claim 2 , wherein the determination of the one or more operations of the ego-machine is further based at least on weighting at least one of the sensor blindness level, the blindness classification, the illumination level, or the visibility distance according to one or more pre-defined weights.

7 . The processor of claim 6 , wherein the sensor blindness level has a highest weight of the one or more pre-defined weights.

8 . The processor of claim 1 , wherein the one or more operations are associated with an operational level, the operation level corresponding to levels of vehicle autonomy including level 0 (L0), level 1 (L1), level 2 (L2), level 3 (L3), level 4 (L4), or level 5 (L5).

9 . The processor of claim 1 , wherein the determination of the one or more operations is further based at least on sensor data generated using one or more other sensor modalities.

10 . The processor of claim 1 , wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

11 . A system comprising:
one or more processing units to:
compute, using one or more neural networks and based at least on image data generated using one or more image sensors:
first data indicating whether a visibility distance associated with the one or more image sensors is reduced, the visibility distance associated with a furthest object discernable using the image data; and
second data indicating a first blindness classification indicating that at least a portion of the image data is associated with blindness and a second blindness classification indicating at least one of a cause or a level for the blindness associated with the at least the portion of the image data;

determine, based at least on the first data and the second data, an operational level of an ego-machine; and
determine one or more control operations based at least on the operational level.

12 . The system of claim 11 , wherein the determination of the operational level is further based at least on third data generated using one or more other sensor modalities.

13 . The system of claim 11 , wherein the determination of the operational level of the ego-machine includes weighting one or more of an illumination level of the one or more sensors, the visibility distance, an amount of blindness of the one or more sensors, the first blindness classification, or the second blindness classification according to one or more pre-defined weights.

14 . The system of claim 13 , wherein the amount of blindness has a highest weight of the one or more pre-defined weights.

15 . The system of claim 11 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

16 . A method comprising:
computing, using one or more neural networks and based at least on image data generated using one or more image sensors of an ego-machine, at least:
first data representative of a visibility distance associated with the one or more image sensors, the visibility distance associated with a distance that one or more objects may be detected using the image data; and
second data representative of a surface condition associated with a road as represented by the image data;
determining, based at least on the first data and the second data, an operational level associated with ego-machine; and controlling an operation of the ego-machine according to the operational level.

17 . The processor of claim 1 , wherein the one or more processing units are further to:
determine, based at least on the image data, one or more parameters, the one or more parameters including at least one of a path surface condition, a scene type classification, or a distance to a scene corresponding to the scene type classification, wherein the determination of the one or more operations of the ego-machine is further based at least on the one or more parameters.

18 . The system of claim 11 , wherein the one or more processing units are further to:
compute, using the one or more neural networks and based at least on the image data generated using the one or more image sensors, third data representative of an amount of blindness associated with the one or more image sensors, wherein the operational level of the ego-machine is further determined based at least on the third data.

19 . The system of claim 11 , wherein the one or more processing units are further to:
compute, using the one or more neural networks and based at least on the image data generated using the one or more image sensors, third data representative of an illumination level, the illumination level associated with an amount of light perceived by a user associated with the ego-machine,

Assignees

• Nvidia Corp

Inventors

• Bajpayee Abhishek
• Gupta Arjun
• Doblar Dylan
• Seo Hae-Jong
• Tang George
• Nagaraja Keerthi Raj

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06F18/213

  Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06V10/82

  using neural networks · CPC title