Trajectory prediction from precomputed or dynamically generated bank of trajectories

Patent metadata

Field	Value
Publication number	US-11912271-B2
Application number	US-202016883899-A
Country	US
Kind code	B2
Filing date	May 26, 2020
Priority date	Nov 7, 2019
Publication date	Feb 27, 2024
Grant date	Feb 27, 2024

Abstract

Among other things, techniques are described for predicting how an agent (e.g., a vehicle, bicycle, pedestrian, etc.) will move in an environment based on prior movement, the road network, the surrounding objects and/or other relevant environmental factors. One trajectory prediction technique involves generating a probability map for an agent's movement. Another trajectory prediction technique involves generating a trajectory lattice, for an agent's movement. In addition, a different trajectory prediction technique involves multi-modal regression where a classifier (e.g., a neural network) is trained to classify the probability of a number of (learned) modes such that each model produces a trajectory based on the current input.

First claim

What is claimed is:

1. A computer implemented method comprising:
receiving, by one or more processors, location data and past trajectory data for one or more objects detected by one or more sensors and training set data comprising a trajectory traveled by an agent;
determining, by the one or more processors, a set of features for the one or more objects based on the location data and the past trajectory data, wherein the set of features comprises information associated with the one or more objects;
combining, by the one or more processors, the set of features, the trajectory traveled by the agent, and motion data of the agent to form a concatenated data set;
generating, by the one or more processors, predicted trajectories based on the concatenated data set, wherein the predicted trajectories are associated with coordinates and corresponding probabilities;
calculating, by the one or more processors, angles between the predicted trajectories and the trajectory traveled by the agent;
selecting, by the one or more processors, a selected trajectory from the predicted trajectories when the angles are within a threshold using a function that selects the selected trajectory from the predicted trajectories based on template trajectories;
calculating, by the one or more processors, a difference between the selected trajectory and the trajectory traveled by the agent using a multi-modal loss function;
adjusting, by the one or more processors, weights of a model based on the difference between the selected trajectory and the trajectory traveled by the agent; and
controlling, by the one or more processors, operation of at least one actuator of a vehicle according to at least one driving command generated based on the model.

2. The method of claim 1 , wherein the templates are fixed templates.

3. The method of claim 1 , wherein the template trajectories are dynamic templates that change based on motion or states of the agent.

4. The method of claim 1 , wherein the difference is calculated by summing a regression loss of the selected trajectory with a classification loss of the predicted trajectories.

5. The method of claim 1 , wherein the weights of a neural network are adjusted by back-propagating the difference through the model.

6. The method of claim 1 , wherein receiving the past trajectory data comprises receiving a trajectory of each object of the one or more object for a past time interval.

7. The method of claim 1 , wherein receiving the location data and the past trajectory data comprises receiving an image including the location data for the one or more objects and the past trajectory data for the one or more objects, wherein the past trajectory data is color coded to indicate a corresponding past trajectory for each object of the one or more objects.

8. A non-transitory computer-readable storage medium comprising one or more programs for execution by one or more processors, the one or more programs including instructions which, when executed by the one or more processors, cause the one or more processors to:
receive location data and past trajectory data for one or more objects detected by one or more sensors and training set data comprising a trajectory traveled by an agent;
determine a set of features for the one or more objects based on the location data and the past trajectory data, wherein the set of features comprises information associated with the one or more objects;
combine the set of features, the trajectory traveled by the agent, and motion data of the agent to form a concatenated data set;
generate predicted trajectories based on the concatenated data set, wherein the predicted trajectories are associated with coordinates and corresponding probabilities;
calculate angles between the predicted trajectories and the trajectory traveled by the agent;
select a selected trajectory from the predicted trajectories when the angles are within a threshold using a function that selects the selected trajectory from the predicted trajectories based on template trajectories;
calculate a difference between the selected trajectory and the trajectory traveled by the agent using a multi-modal loss function;
adjust weights of a model based on the difference between the selected trajectory and the trajectory traveled by the agent; and
control operation of at least one actuator of a vehicle according to at least one driving command generated based on the model.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the templates are fixed templates.

10. The non-transitory computer-readable storage medium of claim 8 , wherein the template trajectories are dynamic templates that change based on motion or states of the agent.

11. The non-transitory computer-readable storage medium of claim 8 , wherein the difference is calculated by summing a regression loss of the selected trajectory with a classification loss of the predicted trajectories.

12. The non-transitory computer-readable storage medium of claim 8 , wherein the weights of a neural network are adjusted by back-propagating the difference through the model.

13. The non-transitory computer-readable storage medium of claim 8 , wherein receiving the past trajectory data comprises receiving a trajectory of each object of the one or more object for a past time interval.

14. A vehicle comprising:
one or more computer-readable media storing computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions to:
receive location data and past trajectory data for one or more objects detected by one or more sensors and training set data comprising a trajectory traveled by an agent;
determine a set of features for the one or more objects based on the location data and the past trajectory data, wherein the set of features comprises information associated with the one or more objects;
combine the set of features, the trajectory traveled by the agent, and motion data of the agent to form a concatenated data set;
generate predicted trajectories based on the concatenated data set, wherein the predicted trajectories are associated with coordinates and corresponding probabilities;
calculate angles between the predicted trajectories and the trajectory traveled by the agent;
select a selected trajectory from the predicted trajectories when the angles are within a threshold using a function that selects the selected trajectory from the predicted trajectories based on template trajectories;
calculate a difference between the selected trajectory and the trajectory traveled by the agent using a multi-modal loss function;
adjust weights of a model based on the difference between the selected trajectory and the trajectory traveled by the agent; and
control operation of at least one actuator of the vehicle according to at least one driving command generated based on the model.

15. The vehicle of claim 14 , wherein the templates are fixed templates.

16. The vehicle of claim 14 , wherein the template trajectories are dynamic templates that change based on motion or states of the agent.

17. The vehicle of claim 14 , wherein the difference is calculated by summing a regression loss of the selected trajectory with a classification loss of the predicted trajectories.

18. The vehicle of claim 14 , wherein the weights of a neural network are adjusted by back-propagating the difference through the model.

19. The vehicle of claim 14 , wherein receiving the past trajectory data comprises receiving a trajectory of each object o

Assignees

• Motional Ad Llc

Inventors

• Phan Tung Minh
• Wolff Eric
• Frazzoli Emilio
• Grigore Elena Corina
• Boulton Freddy

Classifications

• B60W30/0956 Primary

  the prediction being responsive to traffic or environmental parameters · CPC title

• G01S7/4814

  of transmitters alone · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G01S17/931 Primary

  of land vehicles · CPC title

• G01S19/01

  Satellite radio beacon positioning systems transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO · CPC title