Agent trajectory prediction using context-sensitive fusion

Patent metadata

Field	Value
Publication number	US-12420846-B2
Application number	US-202217700348-A
Country	US
Kind code	B2
Filing date	Mar 21, 2022
Priority date	Mar 19, 2021
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using context-sensitive fusion.

First claim

What is claimed is:

1. A method performed by one or more computers, the method comprising:
obtaining context data characterizing an environment, the context data comprising:
data characterizing a plurality of road features in the environment;

for each of one or more target agents of a plurality of agents in the environment:
generating, from the context data, a respective polyline embedding for each of a plurality of polylines that represent the road features, the generating comprising:
for each particular road feature:
obtaining, for each of a plurality of polylines derived from one or more parametric curves that represent the particular road feature in a map of the environment, a respective feature vector that characterizes the polyline; and
processing each of the respective feature vectors using a polyline embedding neural network to generate a respective polyline embedding for each of the plurality of polylines derived from the one or more parametric curves that represent the particular road feature;

generating a contextual embedding for the target agent from at least the respective polyline embeddings; and
generating, from the contextual embedding, a future trajectory prediction for the target agent that characterizes a predicted future trajectory of the target agent after the current time point; and
controlling an autonomous vehicle to follow a planned path based on the future trajectory prediction for the agent.

2. The method of claim 1 , wherein each of the respective feature vectors characterizes a spatial relationship of the target agent to the polyline.

3. The method of claim 1 , wherein the context data further comprises data characterizing trajectories of each of the plurality of agents in the environment at a current time point, wherein the data characterizing the trajectories comprises, for each agent, a respective sequence of observations that each characterize a state of the agent at a different time point, and wherein generating the agent state history embedding comprises:
processing an input sequence derived from the respective sequence of observations for the target agent using an agent state history embedding neural network to generate at least a portion of the agent state history embedding.

4. The method of claim 3 , further comprising generating the input sequence derived from the respective sequence of observations for the target agent by transforming the respective sequence into a reference frame of the target agent.

5. The method of claim 3 , wherein the agent state history embedding neural network is a recurrent neural network and wherein the portion of the agent state history embedding is a hidden state of the recurrent neural network after processing the last observation in the input sequence.

6. The method of claim 3 , wherein the data characterizing the trajectories comprises, for each agent, a respective sequence of observations that each characterize a state of the agent at a different time point, and further comprising generating an aggregated interaction embedding characterizing motion of other agents in the environment relative to the target agent, the generating comprising:
for each of one or more other agents in the environment:
transforming the respective sequence for the other agent into a reference frame of the target agent to generate a respective transformed sequence for the other agent; and
processing the respective transformed sequence for the other agent using an agent embedding neural network to generate an embedding for the other agent; and

combining the embeddings for the other agents to generate the interaction embedding for the target agent.

7. The method of claim 6 , wherein the other agent embedding neural network is a recurrent neural network and wherein the embedding for the other agent is a hidden state of the recurrent neural network after processing the last observation in the transformed sequence.

8. The method of claim 3 , wherein each observation in the sequence includes data indicating whether the observation is a valid observation of the state of the corresponding agent at the corresponding time point or an invalid observation generated because the corresponding agent was not observed in the environment at the corresponding time point.

9. The method of claim 6 , wherein generating the contextual embedding for the target agent from at least the respective polyline embeddings comprises:
generating the contextual embedding for the target agent from the respective polyline embeddings and a agent state history embedding characterizing the trajectory of the target agent; and
generating a fused embedding for the target agent from the contextual embedding, the agent state history embedding, and the interaction embedding, the generating comprising:
concatenating the contextual embedding, an embedding derived from the agent state history embedding, and the interaction embedding.

10. The method of claim 6 , wherein generating the contextual embedding comprises:
initializing a context vector from the agent state history embedding; and
updating the context vector and the polyline embeddings at each of a plurality of iterations.

11. The method of claim 10 , wherein the updating comprises:
at each iteration:
processing the context vector using a first gating neural network to generate a gating context embedding;
for each polyline,
processing the polyline embedding for the polyline using a second gating neural network to generate a gating vector, and
updating the polyline embedding from the gating vector for the polyline and the gating context embedding; and

after updating the polyline embeddings, updating the context vector from the polyline embeddings.

12. The method of claim 11 , wherein the first gating neural network is the same as the second gating neural network.

13. The method of claim 11 , wherein updating the context vector from the polyline embeddings comprises:
combining the polyline embeddings to generate a combined polyline embedding.

14. The method of claim 1 , wherein the future trajectory prediction defines a probability distribution over possible future trajectories for the target agent.

15. The method of claim 14 , wherein the future trajectory prediction comprises a respective trajectory prediction for each of a plurality of anchor trajectories that comprises a probability that the anchor trajectory is the closest anchor trajectory to the future trajectory of the agent.

16. The method of claim 15 , wherein the trajectory prediction for each of the anchor trajectories comprises a deviation output that defines deviations from the anchor trajectory given that the anchor trajectory is the closest anchor trajectory to the future trajectory of the agent.

17. The method of claim 1 , wherein generating the future trajectory prediction output comprises:
for each of a plurality of prediction heads, processing the fused embedding using the prediction head to generate a respective initial trajectory prediction for each of a plurality of anchor trajectories; and
generating the future trajectory prediction output by combining the initial trajectory predictions generated by the plurality of prediction heads.

18. The method of claim 1 , wherein
each of the plurality of agents is an agent in a vicinity of an autonomous vehicle in an environment, and
the context data comprises data generated from data captured by one or more sensors of the autonomous vehicle.

19. The method of claim 1 , further comprising:
providing (i) the trajectory pre

Assignees

• Waymo Llc

Inventors

• Varadarajan Balakrishnan
• Hefny Ahmed Said Mohammed
• Sapp Benjamin
• Refaat Khaled
• Anguelov Dragomir

Classifications

• G05B17/02

  electric · CPC title

• B60W50/0097

  Predicting future conditions · CPC title

• B60W2556/35

  Data fusion · CPC title

• B60W2556/10

  Historical data · CPC title

• G06F18/24133

  Distances to prototypes · CPC title