Agent trajectory prediction using anchor trajectories

Patent metadata

Field	Value
Publication number	US-12097889-B2
Application number	US-202318194882-A
Country	US
Kind code	B2
Filing date	Apr 3, 2023
Priority date	Jul 3, 2019
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

First claim

What is claimed is:

1. A method performed by one or more data processing apparatus, the method comprising:
obtaining an embedding that characterizes an agent in a vicinity of a vehicle in an environment up to a current time point;
processing the embedding using a trajectory prediction neural network to generate a trajectory prediction output that characterizes a future trajectory of the agent after the current time point, wherein:
the trajectory prediction output comprises data characterizing, for each respective anchor trajectory of a plurality of anchor trajectories, a respective predicted similarity between the respective anchor trajectory and the future trajectory of the agent; and
each anchor trajectory characterizes a possible future trajectory of the agent across multiple future time points after the current time point; and

providing the trajectory prediction output to a planning system of the vehicle for controlling the vehicle.

2. The method of claim 1 , wherein the trajectory prediction output further comprises, for each of the plurality of anchor trajectories:
data characterizing, for each waypoint spatial location of the anchor trajectory, a probability distribution dependent on the waypoint spatial location that defines respective likelihoods that the agent will occupy respective spatial positions in a vicinity of the waypoint spatial location at the future time point corresponding to the waypoint spatial location.

3. The method of claim 2 , wherein the data characterizing the probability distribution dependent on the waypoint spatial location comprises data defining parameters of a parametric probability distribution dependent on the waypoint spatial location.

4. The method of claim 3 , wherein the parametric probability distribution dependent on the waypoint spatial location is a Normal probability distribution, and the data defining the parameters of the Normal probability distribution comprise (i) an offset parameter specifying an offset of a mean of the Normal probability distribution from the waypoint spatial location, and (ii) covariance parameters of the Normal probability distribution.

5. The method of claim 2 , wherein the trajectory prediction neural network comprises one or more recurrent neural network layers.

6. The method of claim 1 , wherein the trajectory prediction neural network comprises one or more convolutional neural network layers.

7. The method of claim 1 , wherein obtaining the embedding that characterizes the agent in the vicinity of the vehicle in the environment up to the current time point comprises:
processing an embedding neural network input that characterizes a previous trajectory of the agent in the environment up to the current time point using an embedding neural network to generate an embedding neural network output;
cropping a portion of the embedding neural network output corresponding to the agent; and
determining the embedding that characterizes the agent based on the cropped portion of the embedding neural network output.

8. The method of claim 7 , wherein the embedding neural network input further characterizes trajectories of one or more other agents in the environment up to the current time point.

9. The method of claim 7 , wherein the embedding neural network input further characterizes: (i) dynamic features of the environment comprising traffic light states, and (ii) static features of the environment comprising one or more of: lane connectivity, lane type, stop lines, and speed limit.

10. The method of claim 7 , wherein the embedding neural network input and the embedding neural network output each comprise a respective three-dimensional data representation that characterizes the environment from a top-down perspective.

11. The method of claim 7 , wherein the embedding neural network comprises one or more convolutional neural network layers.

12. The method of claim 7 , wherein determining the embedding that characterizes the agent based on the cropped portion of the embedding neural network output comprises:
rotating the cropped portion of the embedding neural network output to an agent-centric coordinate system.

13. The method of claim 1 , wherein the anchor trajectories are predetermined.

14. The method of claim 13 , wherein pre-determining the anchor trajectories comprises clustering agent trajectories from a training set of agent trajectories.

15. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers,

wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining an embedding that characterizes an agent in a vicinity of a vehicle in an environment up to a current time point;
processing the embedding using a trajectory prediction neural network to generate a trajectory prediction output that characterizes a future trajectory of the agent after the current time point, wherein:
the trajectory prediction output comprises data characterizing, for each respective anchor trajectory of a plurality of anchor trajectories, a respective predicted similarity between the respective anchor trajectory and the future trajectory of the agent; and
each anchor trajectory characterizes a possible future trajectory of the agent across multiple future time points after the current time point; and

providing the trajectory prediction output to a planning system of the vehicle for controlling the vehicle.

16. The system of claim 15 , wherein the trajectory prediction output further comprises, for each of the plurality of anchor trajectories:
data characterizing, for each of waypoint spatial locations of the anchor trajectory, a probability distribution dependent on the waypoint spatial location that defines respective likelihoods that the agent will occupy respective spatial positions in a vicinity of the waypoint spatial location at the future time point corresponding to the waypoint spatial location.

17. The system of claim 16 , wherein the data characterizing the probability distribution dependent on the waypoint spatial location comprises data defining parameters of a parametric probability distribution dependent on the waypoint spatial location.

18. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining an embedding that characterizes an agent in a vicinity of a vehicle in an environment up to a current time point;
processing the embedding using a trajectory prediction neural network to generate a trajectory prediction output that characterizes a future trajectory of the agent after the current time point, wherein:
the trajectory prediction output comprises data characterizing, for each respective anchor trajectory of a plurality of anchor trajectories, a respective predicted similarity between the respective anchor trajectory and the future trajectory of the agent; and
each anchor trajectory characterizes a possible future trajectory of the agent across multiple future time points after the current time point; and

providing the trajectory prediction output to a planning system of the vehicle for controlling the vehicle.

19. The non-transitory computer storage media of claim 18 , wherein the trajectory prediction output further comprises, for each of the plurality of anchor trajectories:
data characterizing, for each of waypoint

Assignees

• Waymo Llc

Inventors

• Chai Yuning
• Sapp Benjamin
• Bansal Mayank
• Anguelov Dragomir

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/82

  using neural networks · CPC title