Planning-aware prediction for control-aware autonomous driving modules

Patent metadata

Field	Value
Publication number	US-12103564-B2
Application number	US-202217570168-A
Country	US
Kind code	B2
Filing date	Jan 6, 2022
Priority date	Jun 17, 2021
Publication date	Oct 1, 2024
Grant date	Oct 1, 2024

Abstract

A method of generating an output trajectory of an ego vehicle includes recording trajectory data of the ego vehicle and pedestrian agents from a scene of a training environment of the ego vehicle. The method includes identifying at least one pedestrian agent from the pedestrian agents within the scene of the training environment of the ego vehicle causing a prediction-discrepancy by the ego vehicle greater than the pedestrian agents within the scene. The method includes updating parameters of a motion prediction model of the ego vehicle based on a magnitude of the prediction-discrepancy caused by the at least one pedestrian agent on the ego vehicle to form a trained, control-aware prediction objective model. The method includes selecting a vehicle control action of the ego vehicle in response to a predicted motion from the trained, control-aware prediction objective model regarding detected pedestrian agents within a traffic environment of the ego vehicle.

First claim

What is claimed is:

1. A method of generating an output trajectory of an ego vehicle, the method comprising:
recording trajectory data of the ego vehicle and pedestrian agents from a scene of a training environment of the ego vehicle;
selecting an nth pedestrian agent in the scene;
enumerating each of the pedestrian agents in the scene other than the nth pedestrian agent;
computing control outputs from a controller of the pedestrian agents according to the recorded trajectory data;
resampling a different trajectory from a known trajectory of the nth agent, ŷ n k ˜q θ (ŷ n |x);
computing a control output from the ego vehicle controller according to the different trajectory û n k =π({ŷ n k }∪y\{y n });
comparing the control output against control outputs of the pedestrian agents according to the recorded trajectory data u=π(y) to identify at least one pedestrian agent causing a prediction-discrepancy by the ego vehicle greater than the pedestrian agents within the scene;
updating parameters of a motion prediction model of the ego vehicle based on a magnitude of the prediction-discrepancy caused by the at least one pedestrian agent on the ego vehicle to form a trained, control-aware prediction objective model; and
selecting a vehicle control action of the ego vehicle in response to a predicted motion from the trained, control-aware prediction objective model regarding detected pedestrian agents within a traffic environment of the ego vehicle.

2. The method of claim 1 , in which updating parameters comprises:
predicting, using the motion prediction model, a future motion of the pedestrian agents and a future motion of the ego vehicle based on the recorded trajectory data; and
computing an attention vector according to the future motion of the pedestrian agents and a future motion of the ego vehicle.

3. The method of claim 2 , in which the updating of parameters comprises:
computing a weighted sum according to the attention vector; and
training the motion prediction model according to the weight sum to learn correlations between planner trajectories and agent trajectories; and
assigning larger attention coefficients to the at least one agent causing the prediction-discrepancy from a controller of the ego vehicle.

4. The method of claim 1 , in which the updating of parameters comprises:
computing a weight for the nth agent according to the control output of the nth agent relative to the control outputs of the pedestrian agents; and
updating the motion prediction model according to the computed weight.

5. The method of claim 1 , further comprising performing the vehicle control action to maneuver the ego vehicle according to the predicted motion of the detected pedestrian agents within the traffic environment of the ego vehicle.

6. The method of claim 1 , in which the vehicle control action comprises throttling, steering, and/or braking.

7. The method of claim 1 , in which updating the parameters comprises training the trained, control-aware prediction objective model to weight a log likelihood of a trajectory of each of the pedestrian agents in the scene by a respective contribution of the pedestrian agents to a control decision of the ego vehicle.

8. A non-transitory computer-readable medium having program code recorded thereon for generating an output trajectory of an ego vehicle, the program code being executed by a processor and comprising:
program code to record trajectory data of the ego vehicle and pedestrian agents from a scene of a training environment of the ego vehicle;
program code to select an nth pedestrian agent in the scene;
program code to enumerate each of the pedestrian agents in the scene other than the nth pedestrian agent;
program code to compute control outputs from a controller of the pedestrian agents according to the recorded trajectory data;
program code to resample a different trajectory from a known trajectory of the nth agent, ŷ n k ˜q θ (ŷ n |x);
program code to compute a control output from the ego vehicle controller according to the different trajectory û n k =π({ŷ n k }∪y\{y n });
program code to compare the control output against control outputs of the pedestrian agents according to the recorded trajectory data u=π(y) to identify at least one pedestrian agent causing a prediction-discrepancy by the ego vehicle greater than the pedestrian agents within the scene;
program code to update parameters of a motion prediction model of the ego vehicle based on a magnitude of the prediction-discrepancy caused by the at least one pedestrian agent on the ego vehicle to form a trained, control-aware prediction objective model; and
program code to select a vehicle control action of the ego vehicle in response to a predicted motion from the trained, control-aware prediction objective model regarding detected pedestrian agents within a traffic environment of the ego vehicle.

9. The non-transitory computer-readable medium of claim 8 , in which the program code to update parameters comprises:
program code to predict, using the motion prediction model, a future motion of the pedestrian agents and a future motion of the ego vehicle based on the recorded trajectory data; and
program code to compute an attention vector according to the future motion of the pedestrian agents and a future motion of the ego vehicle.

10. The non-transitory computer-readable medium of claim 9 , in which the program code to update the parameters comprises:
program code to compute a weighted sum according to the attention vector;
program code to train the motion prediction model according to the weight sum to learn correlations between planner trajectories and agent trajectories; and
program code to assign larger attention coefficients to the at least one agent causing the prediction-discrepancy from a controller of the ego vehicle.

11. The non-transitory computer-readable medium of claim 8 , in which the program code to update the parameters comprises:
program code to compute a weight for the nth agent according to the control output of the nth agent relative to the control outputs of the pedestrian agents; and
program code to update the motion prediction model according to the computed weight.

12. The non-transitory computer-readable medium of claim 8 , further comprising program code to perform the vehicle control action to maneuver the ego vehicle according to the predicted motion of the detected pedestrian agents within the traffic environment of the ego vehicle.

13. The non-transitory computer-readable medium of claim 8 , in which the vehicle control action comprises throttling, steering, and/or braking.

14. The non-transitory computer-readable medium of claim 8 , in which the program code to update the parameters comprises program code to train the trained, control-aware prediction objective model to weight a log likelihood of a trajectory of each of the pedestrian agents in the scene by a respective contribution of the pedestrian agents to a control decision of the ego vehicle.

15. A system for generating an output trajectory of an ego vehicle, the system comprising:
a vehicle perception module to record trajectory data of the ego vehicle and pedestrian agents from a scene of a training environment of the ego vehicle;
a control-aware prediction objective model to select an nth pedestrian agent in the scene, to enumerate each of the pedestrian agents in the scene other than the nth pedestrian agent, to compute control outputs from a controller of the pedestrian agents according to the recorded trajectory data, to resample a different trajectory from a known trajectory of t

Assignees

• Toyota Res Inst Inc
• Univ California

Inventors

• Mcallister Rowan Thomas
• Wulfe Blake Warren
• Mercat Jean
• Ellis Logan Michael
• Levine Sergey
• Gaidon Adrien David

Classifications

• B60W10/04

  including control of propulsion units · CPC title

• B60W2554/4029

  Pedestrians · CPC title

• G05B13/048

  using a predictor · CPC title

• G05B13/0265

  the criterion being a learning criterion · CPC title

• B60W10/20

  including control of steering systems · CPC title