Particle-based hazard detection for autonomous machine

Patent metadata

Field	Value
Publication number	US-12235353-B2
Application number	US-202117454389-A
Country	US
Kind code	B2
Filing date	Nov 10, 2021
Priority date	Nov 10, 2021
Publication date	Feb 25, 2025
Grant date	Feb 25, 2025

Abstract

In various examples, a hazard detection system fuses outputs from multiple sensors over time to determine a probability that a stationary object or hazard exists at a location. The system may then use sensor data to calculate a detection bounding shape for detected objects and, using the bounding shape, may generate a set of particles, each including a confidence value that an object exists at a corresponding location. The system may then capture additional sensor data by one or more sensors of the ego-machine that are different from those used to capture the first sensor data. To improve the accuracy of the confidences of the particles, the system may determine a correspondence between the first sensor data and the additional sensor data (e.g., depth sensor data), which may be used to filter out a portion of the particles and improve the depth predictions corresponding to the object.

First claim

What is claimed is:

1. At least one processor comprising:
processing circuitry to:
detect an object in an environment based at least on first sensor data representative of an image generated at a first time using an image sensor of an ego-machine;
based at least on the detection of the object, determine a first probability distribution corresponding to a location of the object in three dimensional (3D) world space;
generate a set of particles at one or more first locations within the first probability distribution having one or more first confidence values indicative of whether the object exists at the one or more first locations;
generate, at a second time after the first time, the set of particles at one or more second locations based at least on the one or more first locations and motion of the ego-machine between the first time and the second time, the set of particles at the one or more second locations including one or more second confidence values indicative of whether the object exists at the one or more second locations;
correlate the first sensor data with second sensor data representative of one or more depth detections generated using a depth sensor of the ego-machine at a third time;
based at least on the correlation, update the one or more second confidence values of the set of particles to one or more third confidence values indicative of whether the object exists at the one or more second locations; and
filter out one or more of the set of particles having associated one or more third confidence values less than a threshold confidence value to determine a subset of the particles.

2. The at least one processor of claim 1 , wherein the processing circuitry is further to determine one or more second probability distributions associated with the one or more second locations based at least on at least one of: an estimated sensor calibration error between the image sensor and the depth sensor or one or more estimated error of predictions of the motion of the ego-machine.

3. The at least one processor of claim 1 , wherein the processing circuitry is further to correlate the first sensor data with the second sensor data by:
determining a first timestamp corresponding to the first sensor data and a second timestamp corresponding to the second sensor data; and
aligning the depth detections with the image based at least on motion of the ego-machine between the first timestamp and the second timestamp.

4. The at least one processor of claim 3 , wherein the processing circuitry is further to align the depth detections with the image further by using at least one of intrinsic parameters or extrinsic parameters of the image sensor and the depth sensor.

5. The at least one processor of claim 1 , processing circuitry is further to further to perform one or more operations using the subset of the particles.

6. The at least one processor of claim 1 , wherein the processing circuitry is further to correlate the first sensor data with the second sensor data by:
projecting the one or more depth detections to the image as one or more second probability distributions; and
based at least on the projecting, determining at least a set of the one or more depth detections associated with the detection of the object in the image.

7. The at least one processor of claim 6 , wherein the detection of the object is represented using a bounding shape, and the one or more processing units determine processing circuitry determines at least the set of the one or more depth detections associated with the detection of the object in the image by determining which of the one or more second probability distributions are located within the bounding shape.

8. The at least one processor of claim 1 , wherein the second confidence values are different than the first confidence values based at least on an estimated error corresponding to motion of the ego-machine.

9. The at least one processor of claim 1 , wherein the at least one processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

10. A system comprising:
one or more processors to execute operations comprising:
determining, using one or more first particle locations of one or more particles corresponding to an object at a first time, one or more second particle locations of the particles corresponding to the object at a second time, the one or more first particle locations being determined using first sensor data obtained using an image sensor associated with a machine;
determining one or more depth detections from second sensor data obtained using a depth sensor associated with the machine are within a bounding shape corresponding to the object as represented by the first sensor data;
determining, using the one or more depth detections within the bounding shape and the one or more second particle locations, one or more confidence values indicative of whether the object exists at the one or more second particle locations;
determining a location of the object using the one or more confidence values; and
performing one or more control operations corresponding to the machine based at least on the location of the object.

11. The system of claim 10 , wherein the one or more first particle locations and the one or more second particle locations are represented in three-dimensional (3D) world space.

12. The system of claim 10 , wherein the determining the one or more depth detections includes projecting the one or more depth detections from three-dimensional (3D) world space to two-dimensional (2D) image space, and comparing the one or more depth detections to the bounding shape in 2D image space.

13. The system of claim 12 , wherein the one or more depth detections are projected as probability distribution functions.

14. The system of claim 10 , wherein the determining the location of the object includes filtering out one or more of the one or more particles having one or more associated confidences below a threshold.

15. The system of claim 10 , wherein the one or more first particle locations are determined using a probability distribution function generated using a detection of the object in an image generated using the image sensor.

16. The system of claim 10 , wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

17. A method comprising:
generating a set of particles at one or more first locations within one or more probability distributions corresponding to a location of an object detected using first sensor data obtained using an image sensor of an ego-machine and corresponding to a first time;
generating, at a second time after the first ti

Assignees

• Nvidia Corp

Inventors

• Pan Gang
• Pehserl Joachim
• Zhang Dong
• Demiroz Baris Evrim
• Ogden Samuel Rupp
• Choe Tae Eun
• Oh Sangmin

Classifications

• G01S2013/9318

  Controlling the steering · CPC title

• G01S13/865 Primary

  Combination of radar systems with lidar systems · CPC title

• G01S17/931

  of land vehicles · CPC title

• G01S2013/932

  using own vehicle data, e.g. ground speed, steering wheel direction · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title