Agent trajectory prediction using anchor trajectories

Patent metadata

Field	Value
Publication number	US-11618481-B2
Application number	US-202016919872-A
Country	US
Kind code	B2
Filing date	Jul 2, 2020
Priority date	Jul 3, 2019
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

First claim

What is claimed is:

1. A method performed by one or more data processing apparatus, the method comprising:
obtaining an embedding that characterizes an agent in a vicinity of a vehicle in an environment up to a current time point;
processing the embedding using a trajectory prediction neural network to generate a trajectory prediction output that characterizes a future trajectory of the agent after the current time point, wherein:
the trajectory prediction output comprises data specifying, for each respective anchor trajectory of a plurality of anchor trajectories, a respective predicted similarity score that characterizes a likelihood that the respective anchor trajectory is, among the plurality of anchor trajectories, the most similar trajectory to the future trajectory of the agent; and
each anchor trajectory characterizes a possible future trajectory of the agent and comprises data specifying a sequence of multiple waypoint spatial locations in the environment that each correspond to a possible position of the agent at a respective future time point; and

providing the trajectory prediction output to a planning system of the vehicle to generate planning decisions that plan a future trajectory of the vehicle.

2. The method of claim 1 , wherein the trajectory prediction output further comprises, for each of the plurality of anchor trajectories:
data characterizing, for each waypoint spatial location of the anchor trajectory, a probability distribution dependent on the waypoint spatial location that defines respective likelihoods that the agent will occupy respective spatial positions in a vicinity of the waypoint spatial location at the future time point corresponding to the waypoint spatial location.

3. The method of claim 2 , wherein the data characterizing the probability distribution dependent on the waypoint spatial location comprises data defining parameters of a parametric probability distribution dependent on the waypoint spatial location.

4. The method of claim 3 , wherein the parametric probability distribution dependent on the waypoint spatial location is a Normal probability distribution, and the data defining the parameters of the Normal probability distribution comprise (i) an offset parameter specifying an offset of a mean of the Normal probability distribution from the waypoint spatial location, and (ii) covariance parameters of the Normal probability distribution.

5. The method of claim 1 , wherein the trajectory prediction neural network comprises one or more convolutional neural network layers.

6. The method of claim 2 , wherein the trajectory prediction neural network comprises one or more recurrent neural network layers.

7. The method of claim 1 , wherein obtaining the embedding that characterizes the agent in the vicinity of the vehicle in the environment up to the current time point comprises:
processing an embedding neural network input that characterizes a previous trajectory of the agent in the environment up to the current time point using an embedding neural network to generate an embedding neural network output;
cropping a portion of the embedding neural network output corresponding to the agent; and
determining the embedding that characterizes the agent based on the cropped portion of the embedding neural network output.

8. The method of claim 7 , wherein the embedding neural network input further characterizes trajectories of one or more other agents in the environment up to the current time point.

9. The method of claim 7 , wherein the embedding neural network input further characterizes: (i) dynamic features of the environment comprising traffic light states, and (ii) static features of the environment comprising one or more of: lane connectivity, lane type, stop lines, and speed limit.

10. The method of claim 7 , wherein the embedding neural network input and the embedding neural network output each comprise a respective three-dimensional data representation that characterizes the environment from a top-down perspective.

11. The method of claim 7 , wherein the embedding neural network comprises one or more convolutional neural network layers.

12. The method of claim 7 , wherein determining the embedding that characterizes the agent based on the cropped portion of the embedding neural network output comprises:
rotating the cropped portion of the embedding neural network output to an agent-centric coordinate system.

13. The method of claim 1 , wherein the anchor trajectories are predetermined.

14. The method of claim 13 , wherein pre-determining the anchor trajectories comprises clustering agent trajectories from a training set of agent trajectories.

15. A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
obtaining an embedding that characterizes an agent in a vicinity of a vehicle in an environment up to a current time point;
processing the embedding using a trajectory prediction neural network to generate a trajectory prediction output that characterizes a future trajectory of the agent after the current time point, wherein:
the trajectory prediction output comprises data specifying, for each respective anchor trajectory of a plurality of anchor trajectories, a respective predicted similarity score that characterizes a likelihood that the respective anchor trajectory is, among the plurality of anchor trajectories, the most similar trajectory to the future trajectory of the agent; and
each anchor trajectory characterizes a possible future trajectory of the agent and comprises data specifying a sequence of multiple waypoint spatial locations in the environment that each correspond to a respective future time point; and

providing the trajectory prediction output to a planning system of the vehicle to generate planning decisions that plan a future trajectory of the vehicle.

16. The system of claim 15 , wherein the trajectory prediction output further comprises, for each of the plurality of anchor trajectories:
data characterizing, for each of the waypoint spatial locations of the anchor trajectory, a probability distribution dependent on the waypoint spatial location that defines respective likelihoods that the agent will occupy respective spatial positions in a vicinity of the waypoint spatial location at the future time point corresponding to the waypoint spatial location.

17. The system of claim 16 , wherein the data characterizing the probability distribution dependent on the waypoint spatial location comprises data defining parameters of a parametric probability distribution dependent on the waypoint spatial location.

18. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
obtaining an embedding that characterizes an agent in a vicinity of a vehicle in an environment up to a current time point;
processing the embedding using a trajectory prediction neural network to generate a trajectory prediction output that characterizes a future trajectory of the agent after the current time point, wherein:
the trajectory prediction output comprises data specifying, for each respective anchor trajectory of a plurality of anchor trajectories, a respective predicted similarity score that characterizes a likelihood that the respective anchor tra

Assignees

• Waymo Llc

Inventors

• Chai Yuning
• Sapp Benjamin
• Bansal Mayank
• Anguelov Dragomir

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G08G1/0112

  from the vehicle, e.g. floating car data [FCD] · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title