Vehicle taillight recognition based on a velocity estimation

Patent metadata

Field	Value
Publication number	US-12249161-B2
Application number	US-202217732401-A
Country	US
Kind code	B2
Filing date	Apr 28, 2022
Priority date	Apr 28, 2022
Publication date	Mar 11, 2025
Grant date	Mar 11, 2025

Abstract

A method for controlling an ego vehicle in an environment includes associating, by a velocity model, one or more objects within the environment with a respective velocity instance label. The method also includes selectively, by a recurrent network of the taillight recognition system, focusing on a selected region of the sequence of images according to a spatial attention model for a vehicle taillight recognition task. The method further includes concatenating the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. The method still further planning a trajectory of the ego vehicle based on inferring, at a classifier of the taillight recognition system, an intent of each object of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label.

First claim

What is claimed is:

1. A method for controlling an ego vehicle in an environment, comprising:
associating, by a velocity model of a taillight recognition system associated with the ego vehicle, a vehicle within the environment with a velocity instance label, the velocity instance label indicating a speed and direction of the vehicle;
selectively focusing, by a recurrent network of the taillight recognition system, on a first region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task, the first region corresponding to a taillight of the vehicle;
concatenating, at the taillight recognition system, the first region with the velocity instance label of the vehicle to generate a concatenated region label, such that the velocity instance label is associated with the taillight of the vehicle;
inferring, at a classifier of the taillight recognition system, an intent of the vehicle according to a taillight state of the vehicle and the concatenated region label; and
planning a trajectory of the ego vehicle based on inferring the intent of the vehicle.

2. The method of claim 1 , further comprising identifying one or more regions of interest in the sequence of images, wherein the first region is one region of the one or more regions of interest.

3. The method of claim 1 , wherein planning the trajectory includes adjusting the trajectory of the ego vehicle to avoid a collision with the vehicle.

4. The method of claim 1 , further comprising:
generating, via a flow model of the taillight recognition system, a two-dimensional (2D) flow vector for each cell grid of a plurality of cell grids based on a first representation and a second representation of the environment; and
determining the velocity instance label of the vehicle based on the 2D flow vector for each cell grid.

5. The method of claim 4 , further comprising:
obtaining the first representation via a first light detection and ranging (LiDAR) sweep performed at a first time period; and
obtaining the second representation via a second LiDAR sweep performed at a second time period.

6. The method of claim 4 , wherein the flow model is associated with a flow loss, the velocity model is associated with a velocity loss, and the classifier is associated with a classification loss.

7. The method of claim 6 , further comprising training the taillight recognition system in an end-to-end manner to minimize a sum of the flow loss, the velocity loss, and the classification loss.

8. An apparatus for controlling an ego vehicle in an environment, comprising:
at least one processor; and
at least one memory coupled with the at least one processor and storing instructions operable, when executed by the at least one processor, to cause the apparatus to:
associate, by a velocity model of a taillight recognition system associated with the ego vehicle, a vehicle within the environment with a velocity instance label, the velocity instance label indicating a speed and direction of the vehicle;
selectively focus, by a recurrent network of the taillight recognition system, on a first region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task, the first region corresponding to a taillight of the vehicle;
concatenate, at the taillight recognition system, the first region with the velocity instance label of the vehicle to generate a concatenated region label, such that the velocity instance label is associated with the taillight of the vehicle;
infer, at a classifier of the taillight recognition system, an intent of the vehicle according to a taillight state of the vehicle and the concatenated region label; and
plan a trajectory of the ego vehicle based on inferring the intent of the vehicle.

9. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to identify one or more regions of interest in the sequence of images, wherein the first region is one region of the one or more regions of interest.

10. The apparatus of claim 8 , wherein execution of the instructions to plan the trajectory further cause the apparatus to adjust the trajectory of the ego vehicle to avoid a collision with the vehicle.

11. The apparatus of claim 8 , wherein execution of the instructions further cause the apparatus to:
generate, via a flow model of the taillight recognition system, a two-dimensional (2D) flow vector for each cell grid of a plurality of cell grids based on a first representation and a second representation of the environment; and
determine the velocity instance label of the vehicle based on the 2D flow vector for each cell grid.

12. The apparatus of claim 11 , wherein execution of the instructions further cause the apparatus to:
obtain the first representation via a first light detection and ranging (LiDAR) sweep performed at a first time period; and
obtain the second representation via a second LiDAR sweep performed at a second time period.

13. The apparatus of claim 11 , wherein the flow model is associated with a flow loss, the velocity model is associated with a velocity loss, and the classifier is associated with a classification loss.

14. The apparatus of claim 13 , wherein execution of the instructions further cause the apparatus to train the taillight recognition system in an end-to-end manner to minimize a sum of the flow loss, the velocity loss, and the classification loss.

15. A non-transitory computer-readable medium having program code recorded thereon for controlling an ego vehicle in an environment, the program code executed by a processor and comprising:
program code to associate, by a velocity model of a taillight recognition system associated with the ego vehicle, a vehicle within the environment with a velocity instance label, the velocity instance label indicating a speed and direction of the vehicle;
program code to selectively focus, by a recurrent network of the taillight recognition system, on a first region of a sequence of images according to a spatial attention model for a vehicle taillight recognition task, the first region corresponding to a taillight of the vehicle;
program code to concatenate, at the taillight recognition system, the first region with the velocity instance label of the vehicle to generate a concatenated region label, such that the velocity instance label is associated with the taillight of the vehicle;
program code to infer, at a classifier of the taillight recognition system, an intent of the vehicle according to a taillight state of the vehicle and the concatenated region label; and
program code to plan a trajectory of the ego vehicle based on inferring the intent of the vehicle.

16. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises program code to identify one or more regions of interest in the sequence of images, wherein the first region is one region of the one or more regions of interest.

17. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises program code to adjust the trajectory of the ego vehicle to avoid a collision with the vehicle.

18. The non-transitory computer-readable medium of claim 15 , wherein the program code further comprises:
program code to generate, via a flow model of the taillight recognition system, a two-dimensional (2D) flow vector for each cell grid of a plurality of cell grids based on a first representation and a second representation of the environment; and
program code to determine the velocity instance la

Assignees

• Toyota Res Inst Inc
• Toyota Motor Co Ltd

Inventors

• Lee Kuan-Hui
• Ochoa Charles Christopher
• Bhargava Arjun
• Fang Chao

Classifications

• B60W2420/408

  Radar; Laser, e.g. lidar · CPC title

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• B60W2554/4042

  Longitudinal speed · CPC title

• B60W2554/4045

  Intention, e.g. lane change or imminent movement · CPC title