Robust behavior prediction neural networks through non-causal agent based augmentation

Patent metadata

Field	Value
Publication number	US-12461532-B2
Application number	US-202318118696-A
Country	US
Kind code	B2
Filing date	Mar 7, 2023
Priority date	Mar 7, 2022
Publication date	Nov 4, 2025
Grant date	Nov 4, 2025

Abstract

Methods, systems, and apparatus for predicting future trajectories of agents in an environment. In one aspect, a system comprises one or more computers configured to receive a data set comprising multiple training examples. The training examples include scene data comprising respective agent data for multiple agents and a ground truth trajectory for a target agent that represents ground truth motion of the target agent after a corresponding time point. The one or more computers obtain data identifying one or more of the multiple agents as non-causal agents for each training example. A non-causal agent is an agent whose states do not cause the ground truth trajectory for the target agent to change. The one or more computers generate a respective modified training example from each of the multiple training examples.

First claim

The invention claimed is:

1 . A method performed by one or more computers, the method comprising:
receiving a training example identifying (i) respective agent data for each of a plurality of agents in a scene of an environment at a corresponding time point, the plurality of agents comprising a target agent and a set of other agents, and (ii) a ground truth trajectory for the target agent that represents ground truth motion of the target agent after the corresponding time point; obtaining data identifying, as non-causal agents, one or more of the set of other agents identified in the training example; and generating a modified training example by modifying the respective agent data for one or more of the other agents that were identified as non-causal agents, the generating comprising:
removing the respective agent data for a set of non-causal agents, wherein a number of non-causal agents in the set is based on a number of causal agents in the scene.

2 . The method of claim 1 , further comprising:
training a behavior prediction neural network on training data that includes the modified training example.

3 . The method of claim 1 , further comprising:
processing modified scene data comprising the modified agent data in the modified training example using a trained behavior prediction neural network to generate a behavior prediction output for the target agent in the modified training example; determining one or more robustness measures for the trained behavior prediction neural network based on (i) the behavior prediction output for the target agent in the modified training example and (ii) the ground truth trajectory for the target agent in the training example.

4 . The method of claim 3 , further comprising:
processing scene data in the corresponding training example using the trained behavior prediction neural network to generate a behavior prediction output for the target agent in the training example; and determining one or more sensitivity measures for the trained behavior prediction neural network based on (i) the behavior prediction output for the target agent in the modified training example generated by processing the modified training example and (ii) the behavior prediction output for the target agent in the modified training example generated by processing the training example.

5 . The method of claim 4 , further comprising:
determining whether to deploy the trained behavior prediction neural network on-board an autonomous vehicle based at least on the one or more sensitivity measures for the trained behavior prediction neural network.

6 . The method of claim 5 , further comprising:
determining whether to deploy the trained behavior prediction neural network on-board an autonomous vehicle based at least on the one or more robustness measures for the trained behavior prediction neural network.

7 . The method of claim 1 , wherein obtaining data identifying, as non-causal agents, one or more of the set of other agents in the scene for the training example comprises:
obtaining inputs from one or more users labeling agents in the scene as causal or non-causal agents.

8 . The method of claim 1 , wherein obtaining data identifying, as non-causal agents, one or more of the set of other agents in the scene for the training example comprises:
determining that one or more stationary agents in the scene are non-causal agents.

9 . A method performed by one or more computers, the method comprising:
receiving a training example identifying (i) respective agent data for each of a plurality of agents in a scene of an environment at a corresponding time point, the plurality of agents comprising a target agent and a set of other agents, and (ii) a ground truth trajectory for the target agent that represents ground truth motion of the target agent after the corresponding time point; obtaining data identifying, as non-causal agents, one or more of the set of other agents identified in the training example; and generating a modified training example by modifying the respective agent data for one or more of the other agents that were identified as non-causal agents, the generating comprising:
applying perturbations to the respective agent data of one or more of the identified non-causal agents by sampling from a particular distribution for the identified non-causal agent.

10 . The method of claim 9 , further comprising:
training a behavior prediction neural network on training data that includes the modified training example.

11 . The method of claim 9 , further comprising:
processing modified scene data comprising the respective modified agent data in the modified training example using a trained behavior prediction neural network to generate a behavior prediction output for the target agent in the modified training example; and determining one or more robustness measures for the trained behavior prediction neural network based on (i) the behavior prediction output for the target agent in the modified training example and (ii) the ground truth trajectory for the target agent in the training example.

12 . The method of claim 11 , further comprising:
processing scene data in the corresponding training example using the trained behavior prediction neural network to generate a behavior prediction output for the target agent in the training example; and determining one or more sensitivity measures for the trained behavior prediction neural network based on, for each modified training example, (i) the behavior prediction output for the target agent in the modified training example generated by processing the modified training example and (ii) the behavior prediction output for the target agent in the modified training example generated by processing the training example.

13 . The method of claim 12 , further comprising:
determining whether to deploy the trained behavior prediction neural network on-board an autonomous vehicle based at least on the one or more sensitivity measures for the trained behavior prediction neural network.

14 . The method of claim 13 , further comprising:
determining whether to deploy the trained behavior prediction neural network on-board an autonomous vehicle based at least on the one or more robustness measures for the trained behavior prediction neural network.

15 . The method of claim 9 , wherein obtaining data identifying, as non-causal agents, one or more of the set of other agents in the scene for the training example comprises:
obtaining inputs from one or more users labeling agents in the scene as causal or non-causal agents.

16 . The method of claim 9 , wherein obtaining data identifying, as non-causal agents, one or more of the set of other agents in the scene for the training example comprises:
determining that one or more stationary agents in the scene are non-causal agents.

17 . A system comprising:
one or more computers; and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving a training example identifying (i) respective agent data for each of a plurality of agents in a scene of an environment at a corresponding time point, the plurality of agents comprising a target agent and a set of other agents, and (ii) a ground truth trajectory for the target agent that represents ground truth motion of the target agent after the corresponding time point; obtaining data identifying, as non-causal agents, one or more of the set of other agents identified in the training

Assignees

• Waymo Llc

Inventors

• Caine Benjamin James
• Refaat Khaled
• Sapp Benjamin
• Ettinger Scott Morgan
• Chai Wei
• Roelofs Rebecca Dawn
• Sun Liting

Classifications

• B60W60/0027

  using trajectory prediction for other traffic participants · CPC title

• G06V20/54

  of traffic, e.g. cars on the road, trains or boats · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title