Traffic signal understandings and representation for prediction, planning, and control

Patent metadata

Field	Value
Publication number	US-12080161-B2
Application number	US-202217732376-A
Country	US
Kind code	B2
Filing date	Apr 28, 2022
Priority date	Apr 28, 2022
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

A method for vehicle prediction, planning, and control is described. The method includes separately encoding traffic state information at an intersection into corresponding traffic state latent spaces. The method also includes aggregating the corresponding traffic state latent spaces to form a generalized traffic geometry latent space. The method further includes interpreting the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories. The method also includes decoding the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories.

First claim

What is claimed is:

1. A method for vehicle prediction, planning, and control, comprising:
recording, using a monocular camera of an ego vehicle, visual traffic signal state information at an intersection, visual turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs), and traffic state information of the ego vehicle;
separately encoding the visual traffic signal state information, the visual turn light and/or brake light state information, and the traffic state information into corresponding traffic state latent spaces;
aggregating the corresponding traffic state latent spaces to form a generalized traffic geometry latent space;
interpreting the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories;
decoding the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories; and
controlling the ego vehicle to follow a planned trajectory at the intersection in response to the predicted vehicle behavior and the current and future vehicle trajectories based on the traffic flow map without relying on map data or light detection and ranging (LIDAR) or radio detection and ranging (RADAR) sensors.

2. The method of claim 1 , in which separately encoding comprises:
encoding visual traffic signal state information at the intersection into a traffic signal state latent space of the corresponding traffic state latent spaces;
encoding turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs) into a turn/brake light signal state latent space of the corresponding traffic state latent spaces; and
encoding traffic state information of an ego vehicle into an ego state latent space of the corresponding traffic state latent spaces.

3. The method of claim 1 , in which aggregating the corresponding traffic state latent spaces comprises:
predicting a future traffic signal state according to a traffic light latent space of the corresponding traffic state latent spaces; and
predicting a vehicle state according to the future traffic signal state and a turn/brake light state latent space of the corresponding traffic state latent spaces.

4. The method of claim 1 , further comprising training a neural network to predict the vehicle behavior of a vehicle at the intersection using a traffic signal prediction as an input.

5. The method of claim 1 , in which generating the traffic flow map comprises:
aggregating vision inputs of traffic light signals at the intersection during operation of an ego vehicle;
predicting a future state of the traffic light signals at the intersection during operation of the ego vehicle;
processing turn/brake light signals of surrounding autonomous dynamic objects (ADOs) into a turn/brake light state latent space of the corresponding latent spaces;
predicting ADO vehicle behavior at the intersection according to the future state of the traffic light signals and the turn/brake light state latent space; and
generating the traffic flow map according to an observed ADO vehicle behavior and the predicted ADO vehicle behavior at the intersection.

6. The method of claim 1 , further comprising:
decoding the generalized traffic geometry latent space; and
predicting an intention of a vehicle at the intersection.

7. The method of claim 1 , further comprising:
decoding the generalized traffic geometry latent space; and
forecasting a trajectory of a vehicle at the intersection.

8. The method of claim 1 , further comprising planning the planned trajectory of the ego vehicle approaching the intersection according to the vehicle behavior and the traffic flow map based on the current and future vehicle trajectories.

9. A non-transitory computer-readable medium having program code recorded thereon for vehicle prediction, planning, and control, the program code being executed by a processor and comprising:
program code to record, using a monocular camera of an ego vehicle, visual traffic signal state information at an intersection, visual turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs), and traffic state information of an ego vehicle;
program code to separately encode the visual traffic signal state information, the visual turn light and/or brake light state information, and the traffic state information into corresponding traffic state latent spaces;
program code to aggregate the corresponding traffic state latent spaces to form a generalized traffic geometry latent space;
program code to interpret the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories;
program code to decode the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories; and
program code to control the ego vehicle to follow a planned trajectory at the intersection in response to the predicted vehicle behavior and the current and future vehicle trajectories based on the traffic flow map without relying on map data or light detection and ranging (LIDAR) or radio detection and ranging (RADAR) sensors.

10. The non-transitory computer-readable medium of claim 9 , in which the program code to separately encode comprises:
program code to encode visual traffic signal state information at the intersection into a traffic signal state latent space of the corresponding traffic state latent spaces;
program code to encode turn light and/or brake light state information of surrounding autonomous dynamic objects (ADOs) into a turn/brake light signal state latent space of the corresponding traffic state latent spaces; and
program code to encode traffic state information of the ego vehicle into an ego state latent space of the corresponding traffic state latent spaces.

11. The non-transitory computer-readable medium of claim 9 , in which the program code to aggregate the corresponding traffic state latent spaces comprises:
program code to predict a future traffic signal state according to a traffic light latent space of the corresponding traffic state latent spaces; and
program code to predict a vehicle state according to the future traffic signal state and a turn/brake light state latent space of the corresponding traffic state latent spaces.

12. The non-transitory computer-readable medium of claim 9 , further comprising program code to train a neural network to predict the vehicle behavior of a vehicle at the intersection using a traffic signal prediction as an input.

13. The non-transitory computer-readable medium of claim 9 , in which generating the traffic flow map comprises:
program code to aggregate vision inputs of traffic light signals at the intersection during operation of the ego vehicle;
program code to predict a future state of the traffic light signals at the intersection during operation of the ego vehicle;
program code to process turn/brake light signals of surrounding autonomous dynamic objects (ADOs) into a turn/brake light state latent space of the corresponding latent spaces;
program code to predict ADO vehicle behavior at the intersection according to the future state of the traffic light signals and the turn/brake light state latent space; and
program code to generate the traffic flow map according to an observed ADO vehicle behavior and the predicted ADO vehicle behavior at the intersection.

14. The non-transitory computer-readable medium of claim 9 , further comprising:
program code t

Assignees

• Toyota Res Inst Inc
• Toyota Motor Co Ltd

Inventors

• Lee Kuan-Hui
• Ochoa Charles Christopher
• Bhargava Arjun
• Fang Chao
• Chen Kun-Hsin

Classifications

• G08G1/0141 Primary

  for traffic information dissemination · CPC title

• G08G1/0125 Primary

  Traffic data processing · CPC title