Systems and methods for traffic signal light detection

Patent metadata

Field	Value
Publication number	US-10699142-B2
Application number	US-201715492669-A
Country	US
Kind code	B2
Filing date	Apr 20, 2017
Priority date	Apr 20, 2017
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

Systems and methods are provided for analyzing traffic signal lights in order to control an autonomous vehicle. A method includes receiving an image from a camera regarding at least one traffic signal light and receiving data related to the traffic signal light. Neural networks determine location and characteristics of the traffic signal light.

First claim

What is claimed is:

1. A method for analyzing traffic signal lights in order to control an autonomous vehicle, comprising:
receiving, by one or more processors, an image from a camera regarding at least one traffic signal light located within the driving environment of the autonomous vehicle;
receiving, by the one or more processors, data related to probable geometry and position of the traffic signal light located within the image; and
applying a machine learning model that includes neural networks, using the one or more processors, to the received data and to the camera image for determining location and characteristics of the traffic signal light, wherein the determined characteristics include characterization of the traffic signal light with respect to shape and color;
wherein two separate of the neural networks are a localization neural network and a classification neural network to analyze the probable geometry and position of the traffic signal light; wherein the two separate of the neural networks include the localization neural network operating prior to the classification network and providing input to the classification neural network;
wherein control of the autonomous vehicle is based upon the determined location and characteristics of the traffic signal light by the neural networks.

2. The method of claim 1 , wherein the camera image is from a fixed exposure camera.

3. The method of claim 1 , wherein locating the one or more traffic signal lights by using the neural networks results in filtering out false positives.

4. The method of claim 1 , wherein the localization neural network determines the location of the traffic signal light;
wherein the classification neural network determines one or more characteristics associated with the traffic signal light, wherein the characteristics include color and shape.

5. The method of claim 4 , wherein state of the traffic signal light is determined based upon the determined characteristics of the traffic signal light by the classification neural network, wherein the state of the traffic signal light includes determining whether the traffic signal light is indicating a left or right turn.

6. The method of claim 4 , wherein a plurality of traffic signal lights is surveyed to generate three-dimensional position data and three-dimensional orientation data of the surveyed traffic signal lights within a points cloud;
wherein the points cloud is used to determine position of the autonomous vehicle with respect to the traffic signal light located within the driving environment and to generate a window where the traffic signal light within the driving environment is expected to be located by the neural networks.

7. The method of claim 4 ,
wherein region of interest windows are projected around a plurality of traffic signal lights in the camera image where their pixel sizes in the image grows larger as the autonomous vehicle get closer;
wherein the region of interest windows are then scaled to a fixed pixel size to provide that the plurality of traffic signal lights in the camera image has the same size in the scaled camera image irrespective of how far away the autonomous vehicle is, thereby allowing a single detector to be trained at a single scale.

8. The method of claim 7 , wherein the localization neural network outputs a grid of values with respect to the window;
wherein the grid of values provides a confidence indication for each grid cell or for groups of adjacent grid cells indicating how likely each of the grid cells or the groups contain a traffic signal light.

9. The method of claim 8 , wherein the grid cells or the groups of adjacent grid cells with confidence indications that satisfy a pre-selected criteria are used by the classification neural network as candidate locations;
wherein the classification neural network outputs a vector of responses with probabilities for different traffic signal light states.

10. The method of claim 1 , wherein semantic information and state information about linked traffic signal lights and the driving environment are used in determining state of the traffic signal light.

11. A system for analyzing traffic signal lights in order to control an autonomous vehicle, the system comprising:
a computer-readable storage device for storing instructions for performing traffic signal analysis for autonomous vehicle operations; and
one or more data processors configured to execute the instructions to:
receive an image from a camera regarding at least one traffic signal light located within the driving environment of the autonomous vehicle;
receive data related to probable geometry and position of the traffic signal light located within the image; and
apply a machine learning model that includes neural networks to the received data and to the camera image for determining location and characteristics of the traffic signal light, wherein the determined characteristics include characterization of the traffic signal light with respect to shape and color;
wherein two separate of the neural networks are a localization neural network and a classification neural network to analyze the probable geometry and position of the traffic signal light; wherein the two separate of the neural networks include the localization neural network operating prior to the classification network and providing input to the classification neural network;
wherein control of the autonomous vehicle is based upon the determined location and characteristics of the traffic signal light by the neural networks.

12. The system of claim 11 , wherein the camera image is from a fixed exposure camera.

13. The system of claim 11 , wherein locating the one or more traffic signal lights by using the neural networks results in filtering out false positives.

14. The system of claim 11 , wherein the localization neural network determines the location of the traffic signal light;
wherein the classification neural network determines one or more characteristics associated with the traffic signal light, wherein the characteristics include color and shape.

15. The system of claim 14 , wherein state of the traffic signal light is determined based upon the determined characteristics of the traffic signal light by the classification neural network, wherein the state of the traffic signal light includes determining whether the traffic signal light is indicating a left or right turn.

16. The system of claim 14 , wherein a plurality of traffic signal lights is surveyed to generate three-dimensional position data and three-dimensional orientation data of the surveyed traffic signal lights within a points cloud;
wherein the points cloud is used to determine position of the autonomous vehicle with respect to the traffic signal light located within the driving environment and to generate a window where the traffic signal light within the driving environment is expected to be located by the neural networks.

17. The system of claim 14 ,
wherein region of interest windows are projected around a plurality of traffic signal lights in the camera image where their pixel sizes in the image grows larger as the autonomous vehicle get closer;
wherein the region of interest windows are then scaled to a fixed pixel size to provide that the plurality of traffic signal lights in the camera image has the same size in the scaled camera image irrespective of how far away the autonomous vehicle is, thereby allowing a single detector to be trained at a single scale.

18. The system of claim 17 , wherein the localization neural network output

Assignees

• Gm Global Tech Operations Llc

Inventors

• Wisniowski Matthias
• Glaser Stuart
• Minster Gautier

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/584

  of vehicle lights or traffic lights · CPC title

• G06T7/73 Primary

  using feature-based methods · CPC title

• G06F18/24133

  Distances to prototypes · CPC title