Detecting traffic signaling states with neural networks

Patent metadata

Field	Value
Publication number	US-11328519-B2
Application number	US-202016936739-A
Country	US
Kind code	B2
Filing date	Jul 23, 2020
Priority date	Jul 23, 2020
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

Machine-learning models are described detecting the signaling state of a traffic signaling unit. A system can obtain an image of the traffic signaling unit, and select a model of the traffic signaling unit that identifies a position of each traffic lighting element on the unit. First and second neural network inputs are processed with a neural network to generate an estimated signaling state of the traffic signaling unit. The first neural network input can represent the image of the traffic signaling unit, and the second neural network input can represent the model of the traffic signaling unit. Using the estimated signaling state of the traffic signaling unit, the system can inform a driving decision of a vehicle.

First claim

What is claimed is:

1. A method for detecting a signaling state of a traffic signaling unit, comprising:
obtaining an image of the traffic signaling unit;
selecting a model of the traffic signaling unit that identifies a position of each traffic lighting element on the traffic signaling unit;
processing, with a neural network, a first neural network input and a second neural network input to generate an estimated signaling state of the traffic signaling unit, wherein the first neural network input represents the image of the traffic signaling unit, and the second neural network input represents the model of the traffic signaling unit; and
using the estimated signaling state of the traffic signaling unit to inform a driving decision of a vehicle.

2. The method of claim 1 , wherein the estimated signaling state of the traffic signaling unit generated by the neural network includes data specifying, for each traffic lighting element on the traffic signaling unit, a respective lighting state of the traffic lighting element.

3. The method of claim 1 , wherein the respective lighting state of each traffic lighting element is selected from a group comprising an on state, an off state, and a flashing state.

4. The method of claim 3 , wherein the respective lighting state of each traffic lighting element further indicates a color of the traffic lighting element.

5. The method of claim 1 , wherein the model comprises an image depicting a respective representation of each traffic lighting element on a model traffic signaling unit of a type corresponding to the traffic signaling unit, wherein the respective representation of each traffic lighting element identifies a shape and a relative position of the traffic lighting element on the model traffic signaling unit.

6. The method of claim 1 , wherein selecting the model of the traffic signaling unit comprises selecting the model from among a plurality of pre-defined models based on a determined type of the traffic signaling unit, wherein different ones of the plurality of pre-defined models correspond to different types of traffic signaling units.

7. The method of claim 1 , wherein the neural network comprises a recurrent neural network.

8. The method of claim 7 , wherein the recurrent neural network is a long short-term memory (LSTM) neural network.

9. The method of claim 1 , wherein the neural network is further configured to process a third neural network input along with the first neural network input and the second neural network input to generate the estimated signaling state of the traffic signaling unit, wherein the third neural network input identifies a type of the traffic signaling unit.

10. The method of claim 1 , further comprising:
updating a hidden state of the neural network as a result of processing the first neural network input and the second neural network input to generate the estimated signaling state of the traffic signaling unit;
obtaining a second image of the traffic signaling unit; and
processing, with the neural network, and in accordance with the updated hidden state of the neural network, the second neural network input and a third neural network input to generate a second estimated signaling state of the traffic signaling unit, wherein the third neural network input represents the second image of the traffic signaling unit.

11. The method of claim 1 , further comprising obtaining a sequence of images of the traffic signaling unit, each image in the sequence depicting the traffic signaling unit at a different time step of a series of time steps;
wherein the neural network is configured:
to process (i) a first neural network input representing an initial image from the sequence of images that depicts the traffic signaling unit at an initial time step of the series of time steps and (ii) the model, to generate an estimated signaling state of the traffic signaling unit at the initial time step; and
for each particular time step in the series of time steps after the initial time step:
to process (i) a first neural network input representing a respective image from the sequence of images that depicts the traffic signaling unit at the particular time step and (ii) the model, to generate an estimated signaling state of the traffic signaling unit at the particular time step that is based in part on at least one input representing an image from the sequence of images at a time step that precedes the particular time step.

12. The method of claim 1 , wherein using the estimated signaling state of the traffic signaling unit to inform the driving decision of the vehicle comprises:
processing the estimated signaling state of the traffic signaling unit to determine an estimated lane state of a lane in a vicinity of the vehicle; and
generating the driving decision of the vehicle based on the estimated lane state.

13. The method of claim 1 , wherein obtaining the image of the traffic signaling unit comprises:
acquiring, with a camera mounted on the vehicle, an image of an environment of the vehicle that encompasses the traffic signaling unit; and
cropping a portion of the image of the environment to substantially isolate the traffic signaling unit in the image.

14. The method of claim 1 , wherein the vehicle comprises a self-driving car that is operable to drive on roadways fully or semi-autonomously.

15. The method of claim 1 , wherein the vehicle comprises a simulated self-driving car.

16. A computing system for detecting a signaling state of a traffic signaling unit, the computing system comprising:
one or more data processing apparatuses; and
one or more computer-readable media having instructions stored thereon that, when executed by the one or more data processing apparatuses, cause performance of operations comprising:
obtaining an image of the traffic signaling unit;
selecting a model of the traffic signaling unit that identifies a position of each traffic lighting element on the traffic signaling unit;
processing, with a neural network, a first neural network input and a second neural network input to generate an estimated signaling state of the traffic signaling unit, wherein the first neural network input represents the image of the traffic signaling unit, and the second neural network input represents the model of the traffic signaling unit; and
using the estimated signaling state of the traffic signaling unit to inform a driving decision of a vehicle.

17. The computing system of claim 16 , wherein the estimated signaling state of the traffic signaling unit generated by the neural network includes data specifying, for each traffic lighting element on the traffic signaling unit, a respective lighting state of the traffic lighting element.

18. The computing system of claim 16 , wherein the respective lighting state of each traffic lighting element is selected from a group comprising an on state, an off state, and a flashing state.

19. The computing system of claim 18 , wherein the respective lighting state of each traffic lighting element further indicates a color of the traffic lighting element.

20. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more data processing apparatuses, cause performance of operations comprising:
obtaining an image of the traffic signaling unit;
selecting a model of the traffic signaling unit that identifies a position of each traffic lighting element on the traffic signaling unit;
processing, with a neural network, a first neural network

Assignees

• Waymo Llc

Inventors

• Hsiao Edward
• Ouyang Yu
• Yao Maoqing

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06V10/764 Primary

  using classification, e.g. of video objects · CPC title

• G06V20/584 Primary

  of vehicle lights or traffic lights · CPC title