Bulb mask representation for traffic light classification

Patent metadata

Field	Value
Publication number	US-11436842-B2
Application number	US-202016817704-A
Country	US
Kind code	B2
Filing date	Mar 13, 2020
Priority date	Mar 13, 2020
Publication date	Sep 6, 2022
Grant date	Sep 6, 2022

Abstract

Systems and methods are provided for representing a traffic signal device. The method includes receiving a digital image of a traffic signal device that includes one or more traffic signal elements, representing the traffic signal device as a raster image, each traffic signal element of the traffic signal device being represented by a mask corresponding to a location of the traffic signal element on the traffic signal device, representing each mask in a channel in the raster image, providing the raster image as an input to a neural network to classify a state for each of the one or more traffic signal elements, and receiving, from the neural network, a classified raster image, in which the classified raster image includes a plurality of masks, each mask representing a state of one of the one or more traffic signal elements.

First claim

The invention claimed is:

1. A method for representing a traffic signal device, the method comprising:
by a computer vision system of a vehicle, receiving a digital image of a traffic signal device that includes one or more traffic signal elements; and
by a processor:
representing the traffic signal device as a raster image, wherein:
each traffic signal element of the traffic signal device is represented by a mask corresponding to a location of the traffic signal element on the traffic signal device,
each mask is represented in only one of a plurality of channels in the raster image, and
each channel in the raster image is a color channel that indicates a color, feature or both of the traffic signal element represented in the mask;

providing the raster image as an input to a neural network to classify a state of each of the one or more traffic signal elements; and
receiving, from the neural network, a classified raster image, in which the classified raster image includes a plurality of masks, wherein each mask represents the state of one of the one or more traffic signal elements.

2. The method of claim 1 , wherein representing the traffic signal device as a raster image comprises using map data to directly generate the raster image.

3. The method of claim 1 , wherein representing the traffic signal device as a raster image comprises:
generating a digital image from map data; and
converting the digital image to a raster image.

4. The method of claim 1 , wherein the plurality of color channels in the raster image correspond to a plurality of different colors.

5. The method of claim 1 , wherein each traffic signal element of the traffic signal device corresponds to a designated light fixture configured to transmit traffic instructions to one or more drivers.

6. The method of claim 1 , wherein the color, feature, or both of each of the one or more traffic signal elements represented in the color channels comprise one or more of the following:
a green light;
a yellow light;
a red light;
a circular light;
a left arrow light;
a right arrow light;
a forward arrow light;
a light having an arrow in any direction;
a flashing green light;
a flashing yellow light;
a flashing red light;
a U-turn light;
a bicycle light; or
an X-light.

7. The method of claim 1 , wherein each mask corresponds to a discrete region of pixels in the raster image.

8. The method of claim 7 , wherein each discrete region of pixels is rectangular in shape.

9. The method of claim 1 , further comprising:
identifying a face of the traffic signal device, wherein each traffic signal element is located within the face of the traffic signal device.

10. The method of claim 1 , further comprising:
generating a confidence value that a traffic signal element of a traffic signal device correlates to the state; and
if the confidence value is greater than a threshold value, determining that the traffic signal element is in the state.

11. The method of claim 1 , further comprising using a position and shape of each mask in the raster image to identify an angle of the traffic signal device.

12. A system for representing a traffic signal device, the system comprising:
a computer vision system configured to receive a digital image of a traffic signal device that includes one or more traffic signal elements;
a transceiver configured to send and receive digital information; and
a processor and program instructions configured to instruct the processor to:
represent the traffic signal device as a raster image in which:
each traffic signal element of the traffic signal device is represented by a mask corresponding to a location of the traffic signal element on the traffic signal device,
each mask is represented in only one of a plurality of channels in the raster image, and
each channel in the raster image is a color channel that indicates a color, feature or both of the traffic signal element represented in the mask;

provide, using the transceiver, the raster image as an input to a neural network to classify a state of each traffic signal element; and
receive, from the neural network, using the transceiver, a classified raster image, in which the classified raster image includes a plurality of masks, each of which represents the state of one of the traffic signal elements.

13. The system of claim 12 , wherein the program instructions to represent the traffic signal device as a raster image comprise instructions to use map data to directly generate the raster image.

14. The system of claim 12 , wherein the program instructions to represent the traffic signal device as a raster image comprise instructions to generate a digital image from map data and convert the digital image to a raster image.

15. The system of claim 12 , wherein the plurality of color channels in the raster image correspond to a plurality of different colors.

16. The system of claim 12 , wherein the color, feature, or both of each of the one or more traffic signal elements represented in the color channels comprise one or more of the following:
a green light;
a yellow light;
a red light;
a circular light;
a left arrow light;
a right arrow light;
a forward arrow light;
a light having an arrow in any direction;
a flashing green light;
a flashing yellow light;
a flashing red light;
a U-turn light;
a bicycle light; or
an X-light.

17. The system of claim 12 , wherein each mask corresponds to a discrete region of pixels in the raster image.

18. The system of claim 12 , wherein the processor is further configured to:
identify a face of the traffic signal device,
wherein each traffic signal element is located within the face of the traffic signal device.

19. A system for representing a traffic signal device, the system comprising:
a processor;
a computer vision system configured to receive a digital image of a traffic signal device that includes one or more traffic signal elements; and
a non-transitory computer-readable storage medium comprising one or more programming instructions that, when executed, cause the processor to:
represent the traffic signal device as a raster image, wherein:
each traffic signal element of the traffic signal device is represented by a mask corresponding to a location of the traffic signal element on the traffic signal device,
each mask is represented in only one of a plurality of in a channels in the raster image, and
each channel in the raster image is a color channel that indicates a color, feature or both of the traffic signal element represented in the mask;

provide the raster image as an input to a neural network to classify a state of each of the one or more traffic signal elements; and
receive, from the neural network, a classified raster image, in which the classified raster image includes a plurality of masks, wherein each mask represents the state of one of the one or more traffic signal elements.

20. The system of claim 19 , wherein the color, feature, or both of each of the one or more traffic signal elements represented in the color channels comprise one or more of the following:
a green light;
a yellow light;
a red light;
a circular light;
a left arrow light;
a right arrow light;
a forward arrow light;
a light having an arrow in any direction;
a flashing green light;
a flashing yellow light;
a flashing red light;
a U-turn light;
a bicycle light; or
an X-light.

Assignees

• Argo Ai Llc

Inventors

• Hotson Guy
• Kwant Richard L
• Yumer Ersin

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/584 Primary

  of vehicle lights or traffic lights · CPC title

• G06V10/56 Primary

  relating to colour · CPC title

• G06V10/60

  relating to illumination properties, e.g. using a reflectance or lighting model · CPC title