Generative adversarial inverse trajectory optimization for probabilistic vehicle forecasting

Patent metadata

Field	Value
Publication number	US-10739773-B2
Application number	US-201816145555-A
Country	US
Kind code	B2
Filing date	Sep 28, 2018
Priority date	Sep 28, 2017
Publication date	Aug 11, 2020
Grant date	Aug 11, 2020

Abstract

Systems and methods for predicting vehicle behavior includes capturing images of a vehicle in traffic using an imaging device. Future behavior of the vehicle is stochastically modeled using a processing device including an energy-based model stored in a memory of the processing device. The energy-based model includes generating a distribution of possible future trajectories of the vehicle using a generator, sampling the distribution of possible future trajectories according to an energy value of each trajectory in the distribution of possible future trajectories an energy model to determine probable future trajectories, and optimizing parameters of each of the generator and the energy model using an optimizer. A user is audibly alerted with a speaker upon an alert system recognizing hazardous trajectories of the probable future trajectories.

First claim

What is claimed is:

1. A system for vehicle behavior prediction, the system comprising:
an imaging device that captures images of a vehicle in traffic;
a processing device including an energy-based model stored in a memory of the processing device to stochastically model future behavior of the vehicle, the energy-based model including:
a generator that produces a distribution of possible future trajectories of the vehicle;
a energy model that samples the distribution of possible future trajectories according to an energy value of each trajectory in the distribution of possible future trajectories to determine probable future trajectories;
an optimizer that optimizes parameters of each of the generator and the energy model; and

an alert system that recognizes hazardous trajectories of the probable future trajectories and generates and audible alert using a speaker.

2. The system as recited in claim 1 , wherein the generator includes a reactive stochastic policy.

3. The system as recited in claim 1 , wherein the generator includes:
a first multilayer perceptron for modeling static information to generate static predictions;
a recurrent neural network for modeling dynamic information to generate dynamic predictions; and
a second multilayer perceptron for generating a future trajectory from the static predictions and the dynamic predictions.

4. The system as recited in claim 1 , wherein the energy model includes a negative entropy module to generate a log probability density of each trajectory in the distribution of possible future trajectories.

5. The system as recited in claim 4 , wherein the energy-based model trains the generator by:
generator the distribution of possible future trajectories using the generator;
energy sampling the distribution of possible future trajectories using the energy model;
negative entropy sampling the distribution of possible future trajectories using the negative entropy module; and
optimizing generator parameters using gradient descent according to a sum of the negative entropy sampling and the energy sampling.

6. The system as recited in claim 1 , wherein the energy model samples the distribution of possible future trajectories by generating a cost map of energies for each feature in an environment depicted in the images.

7. The system as recited in claim 1 , wherein the imaging device includes an image recognition system to perform semantic segmentation on the images and identify features of an environment depicted in the images.

8. The system as recited in claim 7 , wherein the image recognition system generates a bird's eye view image corresponding to each image captured by the imaging device.

9. The system as recited in claim 1 , wherein the optimizer alternates between updating generator parameters corresponding to the generator and updated energy parameters corresponding to the energy model according to adversarial optimization.

10. The system as recited in claim 1 , wherein the energy-based model trains the energy model by:
generating energies with the energy model for each trajectory in each of the distribution of possible future trajectories and a training distribution; and
updating energy parameters according to adversarial optimization between the energies of the possible future trajectories and the energies of the training distribution.

11. A method for vehicle behavior prediction, the method comprising:
capturing images of a vehicle in traffic using an imaging device;
stochastically modeling future behavior of the vehicle using a processing device including an energy-based model stored in a memory of the processing device, the energy-based model including:
generating a distribution of possible future trajectories of the vehicle using a generator;
sampling the distribution of possible future trajectories according to an energy value of each trajectory in the distribution of possible future trajectories a energy model to determine probable future trajectories;
optimizing parameters of each of the generator and the energy model using an optimizer; and

audibly alerting a user with a speaker upon an alert system recognizing hazardous trajectories of the probable future trajectories.

12. The method as recited in claim 11 , wherein the generator includes a reactive stochastic policy.

13. The method as recited in claim 11 , wherein the generator includes:
a first multilayer perceptron for modeling static information to generate static predictions;
a recurrent neural network for modeling dynamic information to generate dynamic predictions; and
a second multilayer perceptron for generating a future trajectory from the static predictions and the dynamic predictions.

14. The method as recited in claim 11 , wherein the energy model includes a negative entropy module to generate a log probability density of each trajectory in the distribution of possible future trajectories.

15. The method as recited in claim 14 , wherein the energy-based model trains the generator by:
generator the distribution of possible future trajectories using the generator;
energy sampling the distribution of possible future trajectories using the energy model;
negative entropy sampling the distribution of possible future trajectories using the negative entropy module; and
optimizing generator parameters using gradient descent according to a sum of the negative entropy sampling and the energy sampling.

16. The method as recited in claim 11 , wherein the energy model samples the distribution of possible future trajectories by generating a cost map of energies for each feature in an environment depicted in the images.

17. The method as recited in claim 11 , wherein the imaging device includes an image recognition system to perform semantic segmentation on the images and identify features of an environment depicted in the images.

18. The method as recited in claim 17 , wherein the image recognition system generates a bird's eye view image corresponding to each image captured by the imaging device.

19. The method as recited in claim 11 , wherein the optimizer alternates between updating generator parameters corresponding to the generator and updated energy parameters corresponding to the energy model according to adversarial optimization.

20. The method as recited in claim 11 , wherein the energy-based model trains the energy model by:
generating energies with the energy model for each trajectory in each of the distribution of possible future trajectories and a training distribution; and
updating energy parameters according to adversarial optimization between the energies of the possible future trajectories and the energies of the training distribution.

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Vernaza Paul
• Choi Wongun
• Rhinehart Nicholas

Classifications

• B60W60/0023

  in response to energy consumption · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• B60W60/0027 Primary

  using trajectory prediction for other traffic participants · CPC title

• G06V30/1916

  Validation; Performance evaluation · CPC title

• G06V30/19173

  Classification techniques · CPC title