Systems and methods for rear signal identification using machine learning

Patent metadata

Field	Value
Publication number	US-10691962-B2
Application number	US-201715713491-A
Country	US
Kind code	B2
Filing date	Sep 22, 2017
Priority date	Sep 22, 2017
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

System, methods, and other embodiments described herein relate to identifying rear indicators of a nearby vehicle. In one embodiment, a method includes, in response to detecting a nearby vehicle, capturing signal images of a rear portion of the nearby vehicle. The method includes computing a braking state for brake lights of the nearby vehicle that indicates whether the brake lights are presently active by analyzing the signal images according to a brake classifier. The method includes computing a turn state for rear turn signals of the nearby vehicle that indicates which of the turn signals are presently active by analyzing regions of interest from the signal images according to a turn classifier. The brake classifier and the turn classifier are comprised of a convolutional neural network and a long short-term memory recurrent neural network (LSTM-RNN). The method includes providing electronic outputs identifying the braking state and the turn state.

First claim

What is claimed is:

1. A signal identification system for identifying rear indicators of a nearby vehicle, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing:
a monitoring module including instructions that when executed by the one or more processors cause the one or more processors to, in response to detecting the nearby vehicle, capturing signal images of a rear portion of the nearby vehicle; and
an indicator module including instructions that when executed by the one or more processors cause the one or more processors to:
i) compute a braking state for brake lights of the nearby vehicle that indicates whether the brake lights are presently active by analyzing the signal images according to a brake classifier, and
ii) compute a turn state for rear turn signals of the nearby vehicle that indicates which of the rear turn signals are presently active by analyzing regions of interest from the signal images according to a turn classifier,
wherein the brake classifier and the turn classifier are each comprised of a combined network architecture including both a convolutional neural network (CNN) and a long short-term memory recurrent neural network (LSTM-RNN) configured in series with the LSTM-RNN accepting an input that is a final output of the CNN, and
wherein the indicator module includes instructions to provide electronic outputs identifying the braking state and the turn state and to control one or more vehicle systems of a host vehicle in response to the electronic outputs.

2. The signal identification system of claim 1 , wherein the indicator module further includes instructions to:
generate the regions of interest from the signal images by
i) compensating for movement of the nearby vehicle between the signal images to produce flow images from the signal images,
ii) comparing the flow images to generate difference images that indicate areas of changed pixels in the flow images, and
iii) extracting the regions of interest from the difference images.

3. The signal identification system of claim 2 , wherein the indicator module further includes instructions to compensate for the movement by computing the flow images using a scale invariant feature transformation (SIFT) flow algorithm to transform the signal images into the flow images, wherein the regions of interest include a region for a left turn indicator and a region for a right turn indicator of the nearby vehicle, and wherein each of the regions of interest are a separate series of difference images that isolate the left turn indicator and the right turn indicator.

4. The signal identification system of claim 1 , wherein the indicator module further includes instructions to compute the braking state by:
i) extracting image spatial features of the nearby vehicle from the signal images by convolving the signal images into layered spatial features and pooling the layered spatial features over multiple layers of a braking convolutional neural network (CNN) of the brake classifier to generate the image spatial features as an electronic output at a fully connected layer of the braking CNN, and
ii) determining, using the image spatial features, temporal dependencies between the signal images that are indicative of the braking state by recurrently analyzing the image spatial features from the signal images according to a braking long short-term memory recurrent neural network (LSTM-RNN) of the brake classifier that indicates the braking state as a probability that the brake lights are presently active,
wherein the braking CNN and the braking LSTM-RNN are trained to identify the braking state.

5. The signal identification system of claim 1 , wherein the indicator module further includes instructions to compute the turn state by:
i) extracting spatial features from the regions of interest by convolving the regions into layered spatial features and pooling the layered spatial features over multiple layers of a turn convolutional neural network (CNN) of the turn classifier to generate the spatial features as an electronic output at a fully connected layer of the turn CNN, and
ii) determining, using the spatial features and the regions of interest, temporal dependencies that are indicative of the turn state by recurrently analyzing the spatial features from the regions of interest according to a turn long short-term memory recurrent neural network (LSTM-RNN) of the turn classifier that iteratively analyzes the regions of interest in relation to a series of the signal images and indicates the turn state as a probability of a particular dynamic flashing state of the rear turn signals,
wherein the turn CNN and the turn LSTM-RNN are trained to identify the turn state.

6. The signal identification system of claim 1 , wherein the monitoring module includes the instructions to capture the signal images including instructions to capture the signal images as a series over a defined period of time in order to capture temporal changes in the rear turn signals, and wherein the monitoring module further include instructions to detect the nearby vehicle including instructions to control at least a camera sensor to analyze acquire scan images of a surrounding environment and analyze the scan images for a presence of the nearby vehicle.

7. The signal identification system of claim 1 , wherein controlling the one or more vehicle systems of the host vehicle by modifying operating parameters of the one or more vehicle systems in response to the braking state and the turn state.

8. The signal identification system of claim 1 , wherein the signal identification system is embedded within an autonomous driving module of a host vehicle.

9. A non-transitory computer-readable medium storing for identifying rear indicators of a nearby vehicle and including instructions that when executed by one or more processors cause the one or more processors to:
compute a braking state for brake lights of the nearby vehicle that indicates whether the brake lights are presently active by analyzing signal images according to a brake classifier, the signal images being captured of a rear portion of the nearby vehicle; and
compute a turn state for rear turn signals of the nearby vehicle that indicates which of the rear turn signals are presently active by analyzing regions of interest from the signal images according to a turn classifier,
wherein the brake classifier and the turn classifier are each comprised of a combined network architecture including both a convolutional neural network (CNN) and a long short-term memory recurrent neural network (LSTM-RNN) configured in series with the LSTM-RNN accepting an input that is a final output of the CNN;
provide electronic outputs identifying the braking state and the turn state; and
control one or more vehicle systems of a host vehicle in response to the electronic outputs.

10. The non-transitory computer-readable medium of claim 9 , further comprising: instructions to, in response to detecting the nearby vehicle, capture the signal images of a rear portion of the nearby vehicle.

11. The non-transitory computer-readable medium of claim 10 , further comprising instructions to:
generate the regions of interest from the signal images by
i) compensating for movement of the nearby vehicle between the signal images to produce flow images from the signal images,
ii) comparing the flow images to generate difference images that indicate areas of changed pixels in the flow images, and
iii) extracting the regions of interest from the difference images.

12. The non-transitory computer-readable medium of claim 9 , wherein the instructions to compute the brak

Assignees

• Toyota Eng & Mfg North America

Inventors

• Mei Xue
• Nagasaka Naoki
• Lee Kuan-Hui
• Prokhorov Danil V

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V20/584

  of vehicle lights or traffic lights · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title