Traffic signal understandings and representation for prediction, planning, and control

Patent metadata

Field	Value
Publication number	US-12424086-B2
Application number	US-202418770534-A
Country	US
Kind code	B2
Filing date	Jul 11, 2024
Priority date	Apr 28, 2022
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

A method for vehicle prediction, planning, and control is described. The method includes separately encoding traffic state information at an intersection into corresponding traffic state latent spaces. The method also includes aggregating the corresponding traffic state latent spaces to form a generalized traffic geometry latent space. The method further includes interpreting the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories. The method also includes decoding the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories.

First claim

What is claimed is:

1. A method for vehicle prediction, planning, and control, comprising:
aggregating vision inputs of traffic light signals at an intersection during operation of an ego vehicle;
predicting a future state of the traffic light signals at the intersection during operation of the ego vehicle;
processing turn/brake light signals of surrounding autonomous dynamic objects (ADOs) into a turn/brake light state latent space of corresponding traffic state latent spaces;
predicting ADO vehicle behavior at the intersection according to the future state of the traffic light signals and a turn/brake light state latent space;
generating a traffic flow map according to an observed ADO vehicle behavior and the predicted ADO vehicle behavior at the intersection; and
planning a trajectory of the ego vehicle through the intersection in response to a predicted vehicle behavior and current and future vehicle trajectories based on the traffic control map without relying on map data or light detection and ranging (LIDAR) or radio detection and ranging (RADAR) sensors.

2. The method of claim 1 , in which aggregating comprises recording, using a monocular camera of the ego vehicle, visual traffic signal state information at the intersection, visual turn light and/or brake light state information of surrounding ADOs, and a traffic state information of the ego vehicle.

3. The method of claim 1 , in which predicting the future state comprises separately encoding the visual traffic signal state information, a visual turn light and/or brake light state information, and a traffic state information into the corresponding traffic state latent spaces.

4. The method of claim 3 , in which predicting the ADO vehicle behavior comprises:
aggregating the corresponding traffic state latent spaces to form a generalized traffic geometry latent space;
interpreting the generalized traffic geometry latent space to form the traffic flow map including current and future vehicle trajectories; and
decoding the generalized traffic geometry latent space to predict the ADO vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories.

5. The method of claim 3 , in which separately encoding comprises:
encoding visual traffic signal state information at the intersection into a traffic signal state latent space of the corresponding traffic state latent spaces;
encoding turn light and/or brake light state information of the surrounding ADOs into a turn/brake light signal state latent space of the corresponding traffic state latent spaces; and
encoding a traffic state information of the ego vehicle into ego state latent space of the corresponding traffic state latent spaces.

6. The method of claim 4 , in which aggregating the corresponding traffic state latent spaces comprises:
predicting a future traffic signal state according to a traffic light latent space of the corresponding traffic state latent spaces; and
predicting a vehicle state according to the future traffic signal state and the turn/brake light state latent space of the corresponding traffic state latent spaces.

7. The method of claim 4 , further comprising:
decoding the generalized traffic geometry latent space;
predicting an intention of a vehicle at the intersection; and
forecasting a trajectory of a vehicle at the intersection.

8. The method of claim 1 , further comprising training a neural network to predict a vehicle behavior of a vehicle at the intersection using a traffic signal prediction as an input.

9. An apparatus, comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
aggregate vision inputs of traffic light signals at an intersection during operation of an ego vehicle;
predict a future state of the traffic light signals at the intersection during operation of the ego vehicle;
process turn/brake light signals of surrounding autonomous dynamic objects (ADOs) into a turn/brake light state latent space of corresponding traffic state latent spaces;
predict ADO vehicle behavior at the intersection according to the future state of the traffic light signals and a turn/brake light state latent space;
generate a traffic flow map according to an observed ADO vehicle behavior and the predicted ADO vehicle behavior at the intersection; and
plan a trajectory of the ego vehicle through the intersection in response to a predicted vehicle behavior and current and future vehicle trajectories based on the traffic control map without relying on map data or light detection and ranging (LIDAR) or radio detection and ranging (RADAR) sensors.

10. The apparatus of claim 9 , in which to aggregate, the processor is further configured to record, using a monocular camera of the ego vehicle, visual traffic signal state information at the intersection, visual turn light and/or brake light state information of surrounding ADOs, and a traffic state information of the ego vehicle.

11. The apparatus of claim 9 , in which to predict the future state, the processor is further configured to separately encode the visual traffic signal state information, a visual turn light and/or brake light state information, and a traffic state information into the corresponding traffic state latent spaces.

12. The method of claim 11 , in which to predict the ADO vehicle behavior, the processor is further configured to:
aggregate the corresponding traffic state latent spaces to form a generalized traffic geometry latent space;
interpret the generalized traffic geometry latent space to form the traffic flow map including current and future vehicle trajectories; and
decode the generalized traffic geometry latent space to predict the ADO vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories.

13. The apparatus of claim 11 , in which to separately encode the processor is further configured to:
encode visual traffic signal state information at the intersection into a traffic signal state latent space of the corresponding traffic state latent spaces;
encode turn light and/or brake light state information of the surrounding ADOs into a turn/brake light signal state latent space of the corresponding traffic state latent spaces; and
encode a traffic state information of the ego vehicle into ego state latent space of the corresponding traffic state latent spaces.

14. The apparatus of claim 12 , in which to aggregate the corresponding traffic state latent spaces, the processor is further configured to:
predict a future traffic signal state according to a traffic light latent space of the corresponding traffic state latent spaces; and
predict a vehicle state according to the future traffic signal state and the turn/brake light state latent space of the corresponding traffic state latent spaces.

15. The apparatus of claim 12 , in which the processor is further configured to:
decode the generalized traffic geometry latent space;
predict an intention of a vehicle at the intersection; and
forecast a trajectory of a vehicle at the intersection.

16. The apparatus of claim 9 , in which the processor is further configured to train a neural network to predict a vehicle behavior of a vehicle at the intersection using a traffic signal prediction as an input.

Assignees

• Toyota Res Inst Inc
• Toyota Motor Co Ltd

Inventors

• Lee Kuan-Hui
• Ochoa Charles Christopher
• Bhargava Arjun
• Fang Chao
• Chen Kun-Hsin

Classifications

• G08G1/0141 Primary

  for traffic information dissemination · CPC title

• G08G1/0125 Primary

  Traffic data processing · CPC title