Multi-modal multi-agent trajectory prediction

Patent metadata

Field	Value
Publication number	US-12049221-B2
Application number	US-202117540140-A
Country	US
Kind code	B2
Filing date	Dec 1, 2021
Priority date	Dec 1, 2020
Publication date	Jul 30, 2024
Grant date	Jul 30, 2024

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using temporal-spatial interaction predictions.

First claim

What is claimed is:

1. A method performed by one or more computers, the method comprising:
obtaining context data characterizing an environment, the context data comprising data characterizing a plurality of agents in the environment at a current time point, and the context data comprising data characterizing trajectories of each of the plurality of agents at the current time point;
determining, from the context data, one or more interacting pairs of agents, wherein each interacting pair of agents includes two agents from the plurality of agents that are likely to interact with one another during a future time period after the current time point;
determining, from the context data and for each of the interacting pairs of agents, a respective temporal-spatial point prediction for the interacting pair of agents that characterizes (i) a location in the environment at which the pair of agents will likely interact and (ii) a future time at which the pair of agents will likely interact;
generating, from the context data and the respective temporal-spatial point predictions, a respective future trajectory prediction for each of one or more of the agents that characterizes a predicted future trajectory of the agent after the current time point; and
controlling a vehicle based on the respective future trajectory prediction for each of one or more of the agents.

2. The method of claim 1 , further comprising:
generating, from the context data, initial graph data representing an initial graph of the environment, wherein:
the initial graph of the environment comprises a plurality of nodes and edges, each node representing a respective one of the plurality of agents and each edge connecting a respective pair of the nodes, and
the initial graph data comprises respective initial node features for each of the nodes and initial edge features for each of the edges for each of the plurality of agents.

3. The method of claim 2 , wherein:
generating the initial graph data comprises generating the respective initial node features for each of the nodes by processing the context data using a backbone encoder neural network to generate the respective initial node features for each of the plurality of agents.

4. The method of claim 2 , wherein:
generating the initial graph data comprises generating the respective initial edge features by computing features that represent a spatial relationship between the agents represented by the nodes that are connected by the edge.

5. The method of claim 2 , wherein determining, from the context data, one or more interacting pairs of agents comprises:
processing the initial graph data representing the initial graph of the environment using a first graph neural network that is configured to process the initial graph data to generate first updated graph data that comprises first updated edge features for each of the edges; and
determining, from the first updated edge features for each of the edges, the one or more interacting pairs of agents.

6. The method of claim 5 , wherein determining, from the first updated edge features for each of the edges, the one or more interacting pairs of agents comprises:
for each of the edges in the initial graph, processing the first updated edge features for the edge using a first neural network to generate an interaction prediction that predicts a likelihood that the pair of agents represented by the pair of nodes that the edge connects in the graph will interact during the future time period; and
selecting the one or more interacting pairs of agents based on the interaction predictions for the edges in the initial graph.

7. The method of claim 2 , wherein determining, from the context data and for each of the interacting pairs of agents, a respective temporal-spatial point prediction for the interacting pair of agents comprises:
processing the initial graph data using a second graph neural network that is configured to process the initial graph data to generate second updated graph data that comprises second updated edge features for each of the edges; and
determining, from the second updated edge features for the edges that connect nodes that represent one of the interacting pairs of agents, the respective temporal-spatial point prediction for each interacting pair of agents.

8. The method of claim 7 , wherein determining, from the second updated edge features for the edges that connect nodes that represent one of the interacting pairs of agents, a respective temporal-spatial point prediction for each interacting pair of agents comprises, for each interacting pair of agents:
processing the second updated edge features for the edge that connects the nodes representing the interacting pair of agents using a second neural network to generate the respective temporal-spatial point prediction for the interacting pair of agents.

9. The method of claim 2 , wherein determining, from the context data and for each of the interacting pairs of agents, a respective temporal-spatial point prediction for the interacting pair of agents comprises:
processing the initial graph data using a second graph neural network that is configured to process the initial graph data to generate second updated graph data that comprises second updated node features for each of the node; and
for each agent in the interacting pair, determining, from the second updated node features for the node that represents the agent, a respective temporal-spatial point prediction for the agent.

10. The method of claim 9 , wherein determining, from the second updated node features for the node that represents the agent, a respective temporal-spatial point prediction for the agent comprises:
processing the second updated node features for the node that represents the agent using a second neural network to generate the respective temporal-spatial point prediction for the agent.

11. The method of claim 2 , wherein generating, from the context data and the respective temporal-spatial point predictions, a respective future trajectory prediction for each of one or more of the agents that characterizes a predicted future trajectory of the agent after the current time point comprises:
generating, from the initial graph data and the respective temporal-spatial point predictions, third updated graph data that comprises third node features for each of the nodes and third edge features for each of the edges;
processing the third graph data using a third graph neural network that is configured to process the third updated graph data to generate fourth updated graph data that comprises fourth updated node features for each of the nodes; and
generating, from the fourth updated node features for one or more of the nodes, a respective future trajectory prediction for each of the agents represented by the one or more nodes that characterizes a predicted future trajectory of the agent after the current time point.

12. The method of claim 11 , wherein generating the third updated graph data comprises:
for each edge that connects nodes that represent an interacting pair, modifying the initial edge features for the edge based on the respective temporal-spatial point prediction for the interacting pair.

13. The method of claim 12 , wherein modifying the initial edge features comprises:
processing the temporal-spatial point prediction using a fourth neural network to generate interaction features for the edge; and
combining the initial edge features for the edge and the interaction features for the edge to generate the third edge features for the edge.

14. The method of claim 13 , wherein generating the third updated graph data fur

Assignees

• Waymo Llc

Inventors

• Sun Pei
• Zhao Hang
• Mccauley Alexander
• Sapp Benjamin
• Gao Jiyang
• Anguelov Dragomir
• Huang Xin
• Shiarlis Kyriacos Christoforos

Classifications

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• B60W50/0097

  Predicting future conditions · CPC title

• B60W40/04

  Traffic conditions · CPC title

• G06N3/049

  Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs · CPC title