Determining traffic light labels and classification quality from infrastructure signals

Patent metadata

Field	Value
Publication number	US-11935309-B2
Application number	US-202017001999-A
Country	US
Kind code	B2
Filing date	Aug 25, 2020
Priority date	Aug 25, 2020
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

This document discloses methods of training a classifier to identify traffic signal states in images captured be a vehicle. The vehicle can then use the identified states when making movement decisions when traveling in an environment. The system determines that a traffic signal is within a field of view of the camera (i.e., within an image). The system also receives a signal with signal phase and timing data for the traffic signal. The system processes the images to identify an image that includes the traffic signal. The system analyzes the signal data to determine a state of the traffic signal at the time of image capture, labels the image with a label of determined state, and passes the image and a label to a classifier in order to train the classifier.

First claim

The invention claimed is:

1. A method of training a classifier to identify traffic signal states in images, the method comprising:
by a vehicle while traveling in an environment:
using a camera to capture images of the environment,
determining that a traffic signal is within a field of view of the camera, and
saving a plurality of the images to a data store along with a time of capture of each saved image; and

by a processor:
processing the saved images to identify an image that includes the traffic signal, determining the time of capture of the identified image,
receiving signal data, from a roadside unit, that indicates signal phase and timing of the traffic signal,
analyzing the signal data received to determine a state of the traffic signal at the time of capture,
labeling the identified image with a label of determined state, and
passing the identified image and the label to a classifier; and by the classifier, using the identified image and the label to train the classifier.

2. The method of claim 1 , further comprising:
by the processor, processing the saved images to identify a plurality of additional images that each include additional traffic signals;
by the processor, for each of the additional images:
extracting, from the data store, a time of capture for the additional image,
analyzing additional signal data to determine a state of the additional traffic signal in the additional image at the time of capture for the additional image,
labeling the additional image with a label the determined state of the additional traffic signal, and

passing the additional image and its label to the classifier; and
by the classifier, using the additional images and the labels for each of the additional images to further train the classifier.

3. The method of claim 1 further comprising:
by a vehicle, using a camera to capture new images of the environment; and
by the processor:
determining that one of the new images includes a traffic signal, and
using the classifier to determine a state of the traffic signal that is in the determined new image.

4. The method of claim 3 further comprising:
receiving, from a roadside unit, new signal data;
extracting a signal state of the traffic signal from the new signal data;
determining whether the extracted signal state matches the state that the classifier determined; and
if the extracted signal state does not match the state that the classifier determined, using the determined new image, the extracted signal state and the state that the classifier determined to further train the classifier.

5. The method of claim 1 , wherein determining that the traffic signal is within a field of view of the camera comprises, by the processor:
using a global positioning sensor to determine a location of the vehicle;
analyzing map data to identify features of the location that are ahead of the vehicle that are in the field of view of the camera; and
determining that one of the identified features is the traffic signal.

6. The method of claim 1 , wherein determining that the traffic signal is within a field of view of the camera comprises:
receiving, via a transceiver of the vehicle, map data from the roadside unit;
using a global positioning system sensor to determine a location of the vehicle; and correlating the map data to the determined location of the vehicle and a planned path of the vehicle to determine that the traffic signal is in the field of view.

7. The method of claim 1 , wherein:
the processor includes a processor component that is integral with the vehicle: and determining that the traffic signal is within a field of view of the camera comprises detecting the traffic signal when processing the images to identify the image that includes the traffic signal.

8. The method of claim 1 further comprising:
by the processor:
receiving an annotation for the identified image, and
when passing the identified image and the label to the classifier, also passing the annotation to the classifier; and

by the classifier, also using the annotation to train the classifier.

9. The method of claim 1 further comprising, by the processor:
identifying an additional image that includes an additional traffic signal;
determining a time of capture for the additional image;
analyzing additional signal data, from the roadside unit, to determine a state of the additional traffic signal in the additional image at the time of capture for the additional image;
receiving a manually-input label for the additional traffic signal;
determining whether the determined state of the additional traffic signal matches the manually-input label; and
if the determined state of the additional traffic signal matches the manually-input label, passing the additional image and its label to the classifier for use in training the classifier, otherwise not passing the additional image to the classifier.

10. A method of training a classifier of an autonomous vehicle to identify traffic signal states in images, the method comprising:
by a processor:
accessing a data store that contains digital images of an environment along with, for each of the images, a time of capture;
processing a group of the digital images identify an image that includes a traffic signal,
determining the time of capture of the identified image,
receiving, from a roadside unit, signal data that indicates signal phase and timing of the traffic signal, wherein the timing of the signal is indicative of how long a current state of the traffic signal is to persist;
analyzing the signal data received at the time of capture of the identified image to determine a state of the traffic signal of the identified image,
labeling the identified image with a label of determined state, and
passing the identified image and the label to a classifier for an autonomous vehicle; and

by the classifier, using the identified image and the label to train the classifier.

11. The method of claim 10 , further comprising, by the processor:
processing the digital images to identify a plurality of additional images that each include additional traffic signals; and
for each of the additional images:
extracting, from the data store, a time of capture for the additional image,
analyzing additional signal data, from the roadside unit, to determine a state of the additional traffic signal in the additional image at the time of capture for the additional image,
labeling the additional image with a label the determined state of the additional traffic signal, and
passing the additional image and its label to the classifier; and

by the classifier, using the additional images and the labels for each of the additional images to further train the classifier.

12. The method of claim 10 further comprising, by the processor:
receiving new images of the environment;
determining that one of the new images includes a traffic signal; and
using the classifier to determine a state of the traffic signal that is in the determined new image.

13. The method of claim 12 further comprising:
receiving, from the roadside unit, new signal data that was collected when the determined new image was captured;
extracting a signal state from the signal data;
determining whether the extracted signal state matches the state that the classifier determined; and
if the extracted signal state does not match the state that the classifier determined, using the determined new image, the extracted signal state and the state that the classifier determined to further train the classifier.

14. The method of claim 10 , wherein receiving the signal da

Assignees

• Ford Global Tech Llc

Inventors

• Hotson Guy
• Kwant Richard L
• Chekan David James
• Azimi Reza

Classifications

• G06V10/772

  Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries · CPC title

• G08G1/096708

  where the received information might be used to generate an automatic action on the vehicle control · CPC title

• G06V20/584 Primary

  of vehicle lights or traffic lights · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G08G1/0116

  from roadside infrastructure, e.g. beacons · CPC title