Anomaly estimation neural networks for behavior prediction

Patent metadata

Field	Value
Publication number	US-12450469-B1
Application number	US-202217842605-A
Country	US
Kind code	B1
Filing date	Jun 16, 2022
Priority date	Jun 16, 2022
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for anomaly estimation for behavior predictions using a neural network. One of the methods includes receiving data characterizing a scene that includes an agent in an environment. A behavior prediction input generated from the data is processed using a behavior prediction model. The behavior prediction model is configured to process the behavior prediction input to generate a predicted probability distribution over a plurality of possible behaviors for the agent. An anomaly estimation input generated from the data is processed using an anomaly estimation model. The anomaly estimation model is configured to process the anomaly estimation input to generate a prediction error for the predicted probability distribution. The prediction error indicates an error between the predicted probability distribution generated by the behavior prediction model and another predicted probability distribution generated by another behavior prediction model.

First claim

What is claimed is:

1. A method comprising:
receiving data characterizing a scene that includes an agent in an environment;
processing a behavior prediction input generated from the data using a behavior prediction neural network, wherein the behavior prediction neural network is configured to receive the behavior prediction input and to process the behavior prediction input to generate a predicted probability distribution over a plurality of possible behaviors for the agent; and
processing an anomaly estimation input generated from the data characterizing the scene that includes the agent using an anomaly estimation neural network, wherein the anomaly estimation neural network is configured to receive the anomaly estimation input and to process the anomaly estimation input to generate a prediction error for the predicted probability distribution, wherein the prediction error indicates an error between the predicted probability distribution over the plurality of possible behaviors generated by the behavior prediction neural network and another predicted probability distribution over the plurality of possible behaviors that would be generated by another behavior prediction neural network from the same behavior prediction input.

2. The method of claim 1 , wherein the behavior prediction neural network is an on-board behavior prediction neural network that runs on-board an autonomous vehicle, and wherein the other behavior prediction neural network is an off-board behavior prediction neural network that consumes more resources than the on-board behavior prediction neural network.

3. The method of claim 2 , wherein the off-board behavior prediction neural network consumes more computational resources than a compute budget allocated for behavior prediction on-board the autonomous vehicle.

4. The method of claim 1 , wherein the behavior prediction neural network has fewer parameters than the other behavior prediction neural network.

5. The method of claim 1 , wherein the prediction error is smaller than a threshold value, further comprising:
generating a planned trajectory for an autonomous vehicle based on the predicted probability distribution.

6. The method of claim 1 , wherein the anomaly estimation input is generated from the data characterizing the scene that includes the agent and the predicted probability distribution.

7. The method of claim 1 , wherein the prediction error is not smaller than the threshold value, further comprising:
generating a planned trajectory for an autonomous vehicle based on the predicted probability distribution; and
modifying a planned trajectory for an autonomous vehicle to account for the predicted probability distribution possibly being erroneous.

8. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
receiving data characterizing a scene that includes an agent in an environment;
processing a behavior prediction input generated from the data using a behavior prediction neural network, wherein the behavior prediction neural network is configured to receive the behavior prediction input and to process the behavior prediction input to generate a predicted probability distribution over a plurality of possible behaviors for the agent; and
processing an anomaly estimation input generated from the data characterizing the scene that includes the agent using an anomaly estimation neural network, wherein the anomaly estimation neural network is configured to receive the anomaly estimation input and to process the anomaly estimation input to generate a prediction error for the predicted probability distribution, wherein the prediction error indicates an error between the predicted probability distribution over the plurality of possible behaviors generated by the behavior prediction neural network and another predicted probability distribution over the plurality of possible behaviors that would be generated by another behavior prediction neural network from the same behavior prediction input.

9. The system of claim 8 , wherein the behavior prediction neural network is an on-board behavior prediction neural network that runs on-board an autonomous vehicle, and wherein the other behavior prediction neural network is an off-board behavior prediction neural network that consumes more resources than the on-board behavior prediction neural network.

10. The system of claim 9 , wherein the off-board behavior prediction neural network consumes more computational resources than a compute budget allocated for behavior prediction on-board the autonomous vehicle.

11. The system of claim 8 , wherein the behavior prediction neural network has fewer parameters than the other behavior prediction neural network.

12. The system of claim 8 , wherein the prediction error is smaller than a threshold value, the operations further comprising:
generating a planned trajectory for an autonomous vehicle based on the predicted probability distribution.

13. The system of claim 8 , wherein the prediction error is not smaller than the threshold value, the operations further comprising:
generating a planned trajectory for an autonomous vehicle based on the predicted probability distribution; and
modifying a planned trajectory for an autonomous vehicle to account for the predicted probability distribution possibly being erroneous.

14. The system of claim 8 , wherein the anomaly estimation input is generated from the data characterizing the scene that includes the agent and the predicted probability distribution.

15. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving data characterizing a scene that includes an agent in an environment;
processing a behavior prediction input generated from the data using a behavior prediction neural network, wherein the behavior prediction neural network is configured to receive the behavior prediction input and to process the behavior prediction input to generate a predicted probability distribution over a plurality of possible behaviors for the agent; and
processing an anomaly estimation input generated from the data characterizing the scene that includes the agent using an anomaly estimation neural network, wherein the anomaly estimation neural network is configured to receive the anomaly estimation input and to process the anomaly estimation input to generate a prediction error for the predicted probability distribution, wherein the prediction error indicates an error between the predicted probability distribution over the plurality of possible behaviors generated by the behavior prediction neural network and another predicted probability distribution over the plurality of possible behaviors that would be generated by another behavior prediction neural network from the same behavior prediction input.

16. The non-transitory computer-readable media of claim 15 , wherein the behavior prediction neural network is an on-board behavior prediction neural network that runs on-board an autonomous vehicle, and wherein the other behavior prediction neural network is an off-board behavior prediction neural network that consumes more resources than the on-board behavior prediction neural network.

17. The non-transitory computer-readable media of claim 16 , wherein the off-board behavior prediction neural network consumes more computational resources than a compu

Assignees

• Waymo Llc

Inventors

• Refaat Khaled

Classifications

• B60W2050/065

  by reducing the computational load on the digital processor of the control computer · CPC title

• B60W2556/45

  External transmission of data to or from the vehicle · CPC title

• B60W60/0027

  using trajectory prediction for other traffic participants · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title