Structured multi-agent interactive trajectory forecasting

Patent metadata

Field	Value
Publication number	US-12528516-B2
Application number	US-202318335920-A
Country	US
Kind code	B2
Filing date	Jun 15, 2023
Priority date	Jun 15, 2022
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

Methods, systems, and apparatus for generating trajectory predictions for one or more agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents. The one or more computers process the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines multiple possible trajectories for the agent after the current time point and a respective likelihood score for each of the multiple possible future trajectories. The one or more computers can generate graph data based on the respective marginal trajectory predictions, and the one or more computers can process the graph data using a graph neural network to generate a joint trajectory prediction output for the multiple agents in the scene.

First claim

What is claimed is:

1 . A method performed by one or more computers, the method comprising:
obtaining scene context data characterizing a scene in an environment at a current time point, wherein the scene includes a plurality of agents; processing the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines (i) a plurality of possible future trajectories for the agent after the current time point and (ii) a respective likelihood score for each of the plurality of possible future trajectories; generating, based on the respective marginal trajectory predictions for the plurality of agents, graph data representing a graph of the scene that comprises a respective node for each of the plurality of agents and edges that each connect a respective pair of nodes, the generating comprising:
determining whether to connect nodes representing a pair of agents of the plurality of agents with an edge based on respective marginal trajectory predictions for the pair of agents;
processing the graph data using a graph neural network to generate a joint trajectory prediction output for the plurality of agents in the scene; and providing at least one of (i) the joint trajectory prediction output or (ii) data derived from the joint trajectory prediction output to an on-board system of an autonomous vehicle for use in controlling the autonomous vehicle.

2 . The method of claim 1 , wherein:
the scene context data comprises data generated from data captured by one or more sensors of an autonomous vehicle, and the agents are agents in a vicinity of the autonomous vehicle in the environment.

3 . The method of claim 2 , wherein the joint trajectory prediction output is generated on-board the autonomous vehicle.

4 . The method of claim 1 , wherein:
the context data comprises data generated from data that simulates data that would be captured by one or more sensors of an autonomous vehicle in the real-world environment, and the target agent is a simulated agent in a vicinity of the simulated autonomous vehicle in the computer simulation.

5 . The method of claim 4 , further comprising:
providing (i) the joint trajectory prediction output, (ii) data derived from the joint trajectory prediction output, or (iii) both for use in controlling the simulated autonomous vehicle in the computer simulation.

6 . The method of claim 1 , wherein determining whether to connect nodes representing the pair of agents with an edge based on the respective marginal trajectory predictions for the pair of agents comprises:
identifying a respective highest-scoring possible future trajectory for each of the agents in the pair according to the likelihood scores in the respective marginal trajectory prediction for the agent; and determining whether to connect nodes representing the pair of agents with an edge based on a similarity between the respective highest-scoring possible future trajectories.

7 . The method of claim 1 , wherein the graph also includes a node representing a conditioning agent and the future behavior prediction for the agent is fixed to a conditioning future trajectory for the conditioning agent.

8 . The method of claim 7 , wherein the conditioning agent is an autonomous vehicle.

9 . The method of claim 1 , wherein the graph also includes a node representing a target agent and the graph is a star-graph that connects the node representing the target agent to all other nodes in the graph.

10 . The method of claim 9 , wherein the target agent is an autonomous vehicle.

11 . The method of claim 1 , wherein generating, based on the respective marginal trajectory predictions, graph data representing a graph of the scene that comprises a respective node for each of the plurality of agents and edges that each connect a respective pair of nodes comprises:
generating, based on the respective marginal trajectory predictions, respective node features for each of the plurality of nodes and respective edge features for each of the plurality of edges.

12 . The method of claim 11 , wherein generating, based on the respective marginal trajectory predictions, respective node features for each of the plurality of nodes and respective edge features for each of the plurality of edges comprises:
for each of the plurality of nodes, determining a unary potential feature for the node from the likelihood scores in the marginal trajectory prediction for the agent represented by the node.

13 . The method of claim 12 , wherein generating, based on the respective marginal trajectory predictions, respective node features for each of the plurality of nodes and respective edge features for each of the plurality of edges comprises:
for each of the plurality of edges, determining a pairwise potential feature for the edge from the marginal trajectory predictions for the pair of agents represented by the nodes connected by the edge.

14 . The method of claim 13 , wherein determining a pairwise potential feature for the edge from the marginal trajectory predictions for the agents represented by the nodes connected by the edge comprises:
generating transformed trajectories for the first agent in the pair by transforming the predicted future trajectories in the marginal behavior prediction for the first agent into the a coordinate system centered at the second agent in the pair; generating transformed trajectories for the second agent in the pair by transforming the predicted future trajectories in the marginal behavior prediction for the second agent into a coordinate system centered at the first agent in the pair; and generating the pairwise potential feature from the transformed trajectories for the first agent and the transformed trajectories for the second agent.

15 . The method of claim 14 , wherein generating the pairwise potential feature from the transformed trajectories for the first agent and the transformed trajectories for the second agent comprises:
processing each pair of transformed trajectories that includes a first agent trajectory and a second agent trajectory using a potential prediction neural network to generate a respective predicted potential score for the pair.

16 . The method of claim 1 , wherein the joint trajectory prediction output identifies a most likely future trajectory for each agent given the marginal trajectory predictions for the plurality of agents.

17 . The method of claim 16 , wherein the joint trajectory prediction output approximates a joint probability distribution over future trajectories for the plurality of agents.

18 . The method of claim 17 , wherein the joint probability distribution is a joint probability distribution over the possible future trajectories in the marginal trajectory predictions for the plurality of agents.

19 . A system comprising:
one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: obtaining scene context data characterizing a scene in an environment at a current time point, wherein the scene includes a plurality of agents; processing the scene context data using a marginal trajectory prediction neural network to generate a respective marginal trajectory prediction for each of the plurality of agents that defines (i) a plurality of possible future trajectories for the agent after the current time point and (ii) a respective li

Assignees

• Waymo Llc

Inventors

• Luo Wenjie
• Park Cheolho
• Anguelov Dragomir
• Sapp Benjamin

Classifications

• B60W2554/4029

  Pedestrians · CPC title

• B60W2554/4026

  Cycles · CPC title

• B60W2554/4046

  Behavior, e.g. aggressive or erratic · CPC title

• B60W50/0097

  Predicting future conditions · CPC title

• B60W30/18145

  Cornering · CPC title