System and method for vehicle taillight state recognition

Patent metadata

Field	Value
Publication number	US-12073324-B2
Application number	US-202318233802-A
Country	US
Kind code	B2
Filing date	Aug 14, 2023
Priority date	Sep 20, 2017
Publication date	Aug 27, 2024
Grant date	Aug 27, 2024

Abstract

A system and method for taillight signal recognition using a convolutional neural network is disclosed. An example embodiment includes: receiving a plurality of image frames from one or more image-generating devices of an autonomous vehicle; using a single-frame taillight illumination status annotation dataset and a single-frame taillight mask dataset to recognize a taillight illumination status of a proximate vehicle identified in an image frame of the plurality of image frames, the single-frame taillight illumination status annotation dataset including one or more taillight illumination status conditions of a right or left vehicle taillight signal, the single-frame taillight mask dataset including annotations to isolate a taillight region of a vehicle; and using a multi-frame taillight illumination status dataset to recognize a taillight illumination status of the proximate vehicle in multiple image frames of the plurality of image frames, the multiple image frames being in temporal succession.

First claim

What is claimed is:

1. A taillight signal recognition apparatus comprising:
a test image collector configured to collect test images and test videos captured by cameras mounted on test vehicles or installed at test locations;
an image annotator configured to annotate the captured images and videos for vehicle taillight recognition;
a deep learning based feature extractor configured for taillight feature extraction from the annotated images; and
a machine learning based model configured for trajectory level taillight state recognition and for inferring a predicted behavior of a vehicle based on the trajectory level taillight state recognition.

2. The apparatus of claim 1 wherein the taillight state recognition is based on a taillight status indication comprising at least one of: a brake light signal, a left turn signal, a right turn signal, and a hazard light signal.

3. The apparatus of claim 1 wherein the inferred predicted behavior of a vehicle comprises at least one of: a braking or slowing action, a left turn, and a right turn.

4. The apparatus of claim 1 wherein the test images and test videos are captured in a variety of different weather conditions, roadway conditions, and lighting conditions.

5. The apparatus of claim 1 including a classifier supervision dataset containing pairs of image patches or portions of vehicle rear surfaces and their classifications.

6. The apparatus of claim 5 wherein the machine learning based model is trained using the classifier supervision dataset.

7. The apparatus of claim 1 including a temporal smoothness dataset containing pairs of image patches or portions of a same vehicle rear surfaces taken at multiple time increments.

8. The apparatus of claim 7 wherein the machine learning based model is trained using the temporal smoothness dataset.

9. The apparatus of claim 1 wherein the taillight state recognition is based on a taillight status indication from the group consisting of: the taillight is invisible or occluded, the taillight is visible but not illuminated or dark, the taillight is visible and illuminated or bright, and the taillight status is unknown.

10. A method comprising:
collecting test images and test videos captured by cameras mounted on test vehicles or installed at test locations;
annotating the captured images and videos for vehicle taillight recognition;
creating a deep learning based feature extractor for taillight feature extraction from the annotated images; and
creating a machine learning based model for taillight state recognition and for inferring a predicted behavior of a vehicle based on the taillight state recognition.

11. The method of claim 10 wherein the taillight state recognition is based on a taillight status indication comprising at least one of: a brake light signal, a left turn signal, and a right turn signal.

12. The method of claim 10 wherein the inferred predicted behavior of a vehicle comprises at least one of: a left turn and a right turn.

13. The method of claim 10 wherein the test images and test videos are captured in a variety of different weather conditions and lighting conditions.

14. The method of claim 10 including a classifier supervision dataset containing pairs of image patches or portions of vehicle rear surfaces and their classifications.

15. The method of claim 10 including a temporal smoothness dataset containing pairs of image patches or portions of a same vehicle rear surfaces taken at multiple time increments.

16. The method of claim 10 wherein human labelers are used to annotate the captured images.

17. The method of claim 10 wherein the machine learning based model is trained using a combination of a classifier supervision dataset and a temporal smoothness dataset.

18. The method of claim 10 wherein the taillight state recognition is based on a taillight status indication from the group consisting of: the taillight is invisible or occluded, the taillight is visible but not illuminated or dark, the taillight is visible and illuminated or bright, and the taillight status is unknown.

19. A non-transitory computer-readable medium storing instructions which, when executed by at least one processor, cause the at least one processor to:
collect test images and test videos captured by cameras mounted on test vehicles or installed at test locations;
annotate the captured images and videos for vehicle taillight recognition;
create a deep learning based feature extractor for taillight feature extraction from the annotated images; and
create a machine learning based model for taillight state recognition and for inferring a predicted behavior of a vehicle based on the taillight state recognition.

20. The non-transitory computer-readable medium of claim 19 wherein the taillight state recognition is based on a taillight status indication comprising at least one of: a brake light signal, a left turn signal, a right turn signal, and a hazard light signal.

Assignees

• Tusimple Inc

Inventors

• Wang Panqu
• Li Tian

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G05D1/81

  Handing over between on-board automatic and on-board manual control · CPC title

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title