Vehicle lighting state determination

Patent metadata

Field	Value
Publication number	US-11126873-B2
Application number	US-201815982658-A
Country	US
Kind code	B2
Filing date	May 17, 2018
Priority date	May 17, 2018
Publication date	Sep 21, 2021
Grant date	Sep 21, 2021

Abstract

Techniques for determining lighting states of a tracked object, such as a vehicle, are discussed herein. An autonomous vehicle can include an image sensor to capture image data of an environment. Objects such can be identified in the image data as objects to be tracked. Frames of the image data representing the tracked object can be selected and input to a machine learning algorithm (e.g., a convolutional neural network, a recurrent neural network, etc.) that is trained to determine probabilities associated with one or more lighting states of the tracked object. Such lighting states include, but are not limited to, a blinker state(s), a brake state, a hazard state, etc. Based at least in part on the one or more probabilities associated with the one or more lighting states, the autonomous vehicle can determine a trajectory for the autonomous vehicle and/or can determine a predicted trajectory for the tracked object.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
capturing image data of an environment using an image sensor on an autonomous vehicle, the image data comprising a plurality of frames captured over time;
identifying a vehicle in the plurality of the frames as a tracked object in the environment;
extracting, from a first frame of the image data, first cropped image data representing the tracked object at a first time;
extracting, from a second frame of the image data, second cropped image data representing the tracked object at a second time;
inputting, substantially simultaneously, a subset of the image data into an input layer of a machine learning algorithm to cause the machine learning algorithm to perform substantially simultaneous analysis on at least the first cropped image data and the second cropped image data, the inputting, substantially simultaneously, the subset of the image data into the input layer of the machine learning algorithm comprising:
inputting, substantially simultaneously, at least the first cropped image data into a first input of the input layer of the machine learning algorithm and inputting the second cropped image data into a second input of the input layer of the machine learning algorithm;

receiving, from the machine learning algorithm, a probability that the first cropped image data representing the tracked object is associated with a lighting state of the tracked object, wherein the lighting state comprises a left blinker state, a right blinker state, a braking state, and a hazard state and the probability comprises:
a first probability associated with the left blinker state of the tracked object;
a second probability associated with the right blinker state of the tracked object;
a third probability associated with the braking state of the tracked object; and
a fourth probability associated with the hazard state of the tracked object;

generating, based at least in part on the probability, a trajectory for the autonomous vehicle; and
causing the autonomous vehicle to follow the trajectory.

2. The system of claim 1 , wherein the machine learning algorithm is trained using training image data annotated to indicate a ground truth state of the lighting state of the tracked object.

3. The system of claim 1 , the operations further comprising:
determining, based at least in part on the first probability, one or more of:
a fifth probability that the tracked object is a double-parked vehicle;
a sixth probability associated with the tracked object entering a region associated with the autonomous vehicle;
a seventh probability associated with a predicted trajectory of the tracked object; or
a change in a velocity associated with the tracked object.

4. The system of claim 1 , the operations further comprising:
selecting the first frame of the image data and the second frame of the image data based at least in part on a sampling frequency.

5. A method comprising:
capturing image data of an environment using an image sensor, the image data comprising a plurality of frames captured over time;
identifying a vehicle as a tracked object in the image data;
selecting a first frame of the image data representing the tracked object at a first time;
selecting a second frame of the image data representing the tracked object at a second time;
inputting, substantially simultaneously, at least the first frame into a first input of an input layer of a machine learning algorithm and the second frame into a second input of the input layer of the machine learning algorithm to cause the machine learning algorithm to perform substantially simultaneous analysis on at least the first frame and the second frame; and
receiving, from the machine learning algorithm, a probability associated with the tracked object indicative of a lighting state of the tracked object.

6. The method of claim 5 , wherein the image sensor is disposed on an autonomous vehicle, the method further comprising:
generating, based at least in part on the probability, a trajectory for the autonomous vehicle; and
causing the autonomous vehicle to follow the trajectory.

7. The method of claim 5 , wherein the lighting state comprises one or more of:
a front left blinker state;
a front right blinker state;
a rear left blinker state;
a rear right blinker state;
a braking state;
a backup state;
an emergency state; or
a hazard state.

8. The method of claim 5 , wherein the probability comprises one or more of:
a first probability associated with a left blinker state of the tracked object;
a second probability associated with a right blinker state of the tracked object;
a third probability associated with a braking state of the tracked object; or
a fourth probability associated with a hazard state of the tracked object.

9. The method of claim 5 further comprising:
receiving, from the machine learning algorithm, a second probability associated with the tracked object indicative of a manual signaling state of the tracked object.

10. The method of claim 5 , wherein the machine learning algorithm comprises at least one of a convolutional neural network or a recurrent neural network.

11. The method of claim 5 , further comprising:
selecting, based at least in part on a sampling frequency, a subset of frames of the plurality of frames as input image data to the machine learning algorithm,
wherein a number of frames of the subset of frames and the sampling frequency is based at least in part on a frame rate of the image sensor.

12. The method of claim 5 , further comprising:
extracting, as extracted image data, a portion of the first frame representing the tracked object;
comparing a size of the extracted image data to an input data size associated with the machine learning algorithm; and
scaling the extracted image data based at least in part on the input data size.

13. The method of claim 5 , further comprising:
selecting, based at least in part on the probability, one of a plurality of predicted trajectories for the tracked object.

14. The method of claim 5 , wherein the image sensor is disposed on an autonomous vehicle, the method further comprising:
determining, based at least in part on the probability, one or more of:
a first probability that the tracked object is a double-parked vehicle;
a second probability associated with the tracked object entering a region associated with the autonomous vehicle;
a third probability associated with a predicted trajectory of the tracked object; or
a change in a velocity associated with the tracked object.

15. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising:
receiving image data of an environment captured by an image sensor on an autonomous vehicle, the image data comprising a plurality of frames captured over time;
identifying a vehicle as a tracked object in the image data;
selecting a first frame of the image data representing the tracked object at a first time;
selecting a second frame of the image data representing the tracked object at a second time;
inputting, substantially simultaneously, at least the first frame into a first input of an input layer of a machine learning algorithm and the second frame into a second input of the input layer of the machine learning algorithm; and
receiving, from the machine lear

Assignees

• Zoox Inc

Inventors

• Lee Tencia
• Wang Kai Zhenyu
• Philbin James William Vaisey

Classifications

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W60/0027

  using trajectory prediction for other traffic participants · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title