Systems and methods for vehicle light signal classification

Patent metadata

Field	Value
Publication number	US-12014549-B2
Application number	US-202117192443-A
Country	US
Kind code	B2
Filing date	Mar 4, 2021
Priority date	Mar 4, 2021
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

A vehicle light classification system captures a sequence of images of a scene that includes a front/rear view of a vehicle with front/rear-side lights, determines semantic keypoints, in the images and associated with the front/rear-side lights, based on inputting the images into a first neural network, obtains multiple difference images that are each a difference between successive images from among the sequence of images, the successive images being aligned based on their respective semantic keypoints, and determines a classification of the front/rear-side lights based at least in part on the difference images by inputting the difference images into a second neural network.

First claim

What is claimed is:

1. A vehicle light classification system, comprising:
a processor; and
a memory communicably coupled to the processor and storing:
a semantic keypoint module including instructions that when executed by the processor cause the processor to: 1) receive, from an image capture device, a sequence of images of a scene that includes a view of a vehicle with lights, and 2) determine, based on inputting the images into a first neural network, both semantic keypoints, in the images and associated with the lights, and classifications for the semantic keypoints wherein a semantic keypoint, of the semantic keypoints, corresponds to a single pixel near a center and away from an edge of an object in an image of the images and the first neural network is a deep neural network; and
a classification module including instructions that when executed by the processor cause the processor to: 1) obtain difference images that are each a difference between successive images from among the sequence of images, the successive images being aligned based on their respective semantic keypoints, and 2) determine a classification of the lights based at least in part on the difference images by inputting the difference images into a second neural network that generates a feature vector.

2. The vehicle light classification system of claim 1 , wherein the classification module further includes instructions that cause the processor to obtain the difference images by modifying, for each difference image, a first image by warping the first image to align iterations of the semantic keypoints across the first image and a successive image, and obtaining a difference between the first image as modified and the successive image.

3. The vehicle light classification system of claim 1 , wherein the second neural network includes a convolutional neural network layer configured to generate feature vectors based on the difference images and a long short-term memory layer configured to generate the classification based at least in part on the feature vectors.

4. The vehicle light classification system of claim 1 , further comprising a tracking module including instructions that when executed by the processor cause the processor to define a bounding box for the vehicle based on the semantic keypoints and track the bounding box across the sequence of images.

5. The vehicle light classification system of claim 4 , wherein the classification module further includes instructions that cause the processor to obtain the difference images based on portions of the images that fall within the bounding box as tracked across the sequence of images.

6. The vehicle light classification system of claim 4 , wherein the scene includes a plurality of vehicles and the instructions to define the bounding box include instructions to define separate bounding boxes for each of the plurality of vehicles based on the semantic keypoints and the instructions to track the bounding box include instructions to track each separate bounding box across the sequence of images.

7. The vehicle light classification system of claim 4 , wherein the instructions to track the bounding box include instructions to compare feature vectors of images in the sequence of images.

8. The vehicle light classification system of claim 1 , wherein the first neural network is configured to output confidence heatmaps that indicate a confidence level of a classification state for the semantic keypoint of the semantic keypoints.

9. The vehicle light classification system of claim 1 , wherein the instructions to determine the semantic keypoints include instructions to select the semantic keypoint that has a confidence level that exceeds a predetermined threshold.

10. A method of classifying vehicle lights, comprising:
capturing a sequence of images of a scene that includes a rear view of a vehicle with lights;
determining, based on inputting the images into a first neural network, both semantic keypoints, in the images and associated with the lights, and classifications for the semantic keypoints, wherein a semantic keypoint, of the semantic keypoints, corresponds to a single pixel near a center and away from an edge of an object in an image of the images and the first neural network is a deep neural network;
obtaining difference images that are each a difference between successive images from among the sequence of images, the successive images being aligned based on their respective semantic keypoints; and
determining a classification of the lights based at least in part on the difference images by inputting the difference images into a second neural network that generates a feature vector.

11. The method of claim 10 , wherein the obtaining the difference images comprises:
modifying a first image by warping the first image to align iterations of the semantic keypoints across the first image and a successive image; and
obtaining a difference between the first image as modified and the successive image.

12. The method of claim 10 , wherein the second neural network includes a convolutional neural network layer configured to generate feature vectors based on the difference images and a long short-term memory layer configured to generate the classification based at least in part on the feature vectors.

13. The method of claim 10 , further comprising:
defining a bounding box for the vehicle based on the semantic keypoints; and
tracking the bounding box across the sequence of images.

14. The method of claim 13 , further comprising obtaining the difference images based on portions of the images that fall within the bounding box as tracked across the sequence of images.

15. The method of claim 13 , wherein:
the scene includes a plurality of vehicles,
the defining the bounding box includes defining separate bounding boxes for each of the plurality of vehicles based on the semantic keypoints, and
the tracking the bounding box includes tracking each separate bounding box across the sequence of images.

16. A non-transitory computer-readable medium for classifying vehicle lights; including instructions that, when executed by one or more processors, cause the one or more processors to:
capture a sequence of images of a scene that includes a front/rear view of a vehicle with lights;
determine, based on inputting the images into a first neural network, both semantic keypoints, in the images and associated with the lights, and classifications for the semantic keypoints, wherein a semantic keypoint, of the semantic keypoints, corresponds to a single pixel near a center and away from an edge of an object in an image of the images and the first neural network is a deep neural network;
obtain difference images that are each a difference between successive images from among the sequence of images, the successive images being aligned based on their respective semantic keypoints; and
determine a classification of the lights based at least in part on the difference images by inputting the difference images into a second neural network that generates a feature vector.

17. The non-transitory computer-readable medium of claim 16 , wherein the instructions to obtain the difference images include instructions to obtain the difference images by:
modifying a first image by warping the first image to align iterations of the semantic keypoints across the first image and a successive image; and
obtaining a difference between the first image as modified and the successive image.

18. The non-transitory computer-readable medium of claim 16 , wherein the second n

Assignees

• Toyota Res Inst Inc

Inventors

• Pan Jia-En
• Lee Kuan-Hui
• Fang Chao
• Chen Kun-Hsin
• Bhargava Arjun
• Pillai Sudeep

Classifications

• G06T3/18

  Image warping, e.g. rearranging pixels individually · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• B60R2300/303

  using joined images, e.g. multiple camera images · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title