Systems and methods for sensor data processing and object detection and motion prediction for robotic platforms

Patent metadata

Field	Value
Publication number	US-12259694-B2
Application number	US-202418656210-A
Country	US
Kind code	B2
Filing date	May 6, 2024
Priority date	Oct 14, 2020
Publication date	Mar 25, 2025
Grant date	Mar 25, 2025

Abstract

Systems and methods are disclosed for detecting and predicting the motion of objects within the surrounding environment of a system such as an autonomous vehicle. For example, an autonomous vehicle can obtain sensor data from a plurality of sensors comprising at least two different sensor modalities (e.g., RADAR, LIDAR, camera) and fused together to create a fused sensor sample. The fused sensor sample can then be provided as input to a machine learning model (e.g., a machine learning model for object detection and/or motion prediction). The machine learning model can have been trained by independently applying sensor dropout to the at least two different sensor modalities. Outputs received from the machine learning model in response to receipt of the fused sensor samples are characterized by improved generalization performance over multiple sensor modalities, thus yielding improved performance in detecting objects and predicting their future locations, as well as improved navigation performance.

First claim

What is claimed is:

1. A computer-implemented method comprising:
(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) applying independent sensor dropout to the at least two different sensor modalities;
(c) fusing the sensor data from the at least two different sensor modalities with sensor dropout independently applied thereto to generate a fused sensor sample;
(d) generating a training data set comprising the fused sensor sample; and
(e) training a machine-learned model for object detection using the training data set, wherein the trained machine-learned model is employed by a robotic platform operating within an environment.

2. The computer-implemented method of claim 1 , wherein the robotic platform comprises an autonomous vehicle.

3. The computer-implemented method of claim 1 , wherein the environment comprises a real-world environment or a simulated environment.

4. The computer-implemented method of claim 1 , wherein the trained machine-learned model comprises an end-to-end model that is configured to jointly perform object detection and motion prediction.

5. The computer-implemented method of claim 1 , wherein (b) comprises independently applying sensor dropout to each of the at least two different sensor modalities at a fixed probability associated with the sensor modality.

6. The computer-implemented method of claim 1 , wherein the plurality of sensors comprise a RADAR system, a LIDAR system, and a camera.

7. The computer-implemented method of claim 6 , wherein:
the at least two different sensor modalities comprise at least one of the RADAR system or the camera; and
(b) comprises zeroing out a final feature vector for a portion of the sensor data obtained from the at least one of the RADAR system or the camera.

8. The computer-implemented method of claim 6 , wherein:
the at least two different sensor modalities comprise the LIDAR system; and
(b) comprises replacing a LIDAR intensity value with a sentinel value for a portion of the sensor data obtained from the LIDAR system.

9. The computer-implemented method of claim 1 , wherein (e) comprises:
inputting the fused sensor sample into the machine-learned model;
generating a loss metric for the machine-learned model based on output of at least a portion of the machine-learned model in response to the fused sensor sample as input; and
modifying at least a portion of the machine-learned model based on the loss metric.

10. The computer-implemented method of claim 9 , wherein the loss metric comprises at least one of a regression loss, a classification loss, an adversarial loss, a multi-task loss, or a perceptual loss.

11. The computer-implemented method of claim 9 , wherein the loss metric is associated with a plurality of loss terms, the plurality of loss terms comprising at least a first loss term associated with a determination or generation of bounding shapes.

12. The computer-implemented method of claim 11 , the plurality of loss terms further comprising at least a second loss term associated with a classification of features.

13. A training computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable medium storing instructions that when executed by the one or more processors cause the training computing system to perform operations, the operations comprising:
(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) applying independent sensor dropout to the at least two sensor modalities;
(c) fusing the sensor data from the at least two different sensor modalities with sensor dropout independently applied thereto to generate a fused sensor sample;
(d) generating a training data set comprising the fused sensor sample; and
(e) training a machine-learned model for object detection using the training data set, wherein the trained machine-learned model is employed by a robotic platform operating within an environment.

14. The training computing system of claim 13 , wherein the robotic platform comprises an autonomous vehicle.

15. The training computing system of claim 13 , wherein the environment comprises a real-world environment or a simulated environment.

16. The training computing system of claim 13 , wherein the trained machine-learned model comprises an end-to-end model that is configured to jointly perform object detection and motion prediction.

17. The training computing system of claim 13 , wherein (b) comprises independently applying sensor dropout to each of the at least two different sensor modalities at a fixed probability associated with the sensor modality.

18. The training computing system of claim 13 , wherein (e) comprises:
inputting the fused sensor sample into the machine-learned model;
generating a loss metric for the machine-learned model based on output of at least a portion of the machine-learned model in response to the fused sensor sample as input; and
modifying at least a portion of the machine-learned model based on the loss metric.

19. The training computing system of claim 18 , wherein the loss metric comprises one or more of a regression loss and a classification loss.

20. One or more non-transitory computer-readable medium storing instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
(a) obtaining sensor data from a plurality of sensors comprising at least two different sensor modalities;
(b) applying independent sensor dropout to the at least two sensor modalities;
(c) fusing the sensor data from the at least two different sensor modalities with sensor dropout independently applied thereto to generate a fused sensor sample;
(d) generating a training data set comprising the fused sensor sample; and
(e) training a machine-learned model for object detection using the training data set, wherein the trained machine-learned model is employed by a robotic platform operating within an environment.

Assignees

• Aurora Operations Inc

Inventors

• Mohta Abhishek
• Chou Fang-Chieh
• Vallespi-Gonzalez Carlos
• Becker Brian C
• Djuric Nemanja

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title