Vehicle intent prediction neural network

Patent metadata

Field	Value
Publication number	US-11480963-B2
Application number	US-201916723787-A
Country	US
Kind code	B2
Filing date	Dec 20, 2019
Priority date	Dec 20, 2019
Publication date	Oct 25, 2022
Grant date	Oct 25, 2022

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating vehicle intent predictions using a neural network. One of the methods includes obtaining an input characterizing one or more vehicles in an environment; generating, from the input, features of each of the vehicles; and for each of the vehicles: processing the features of the vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent from a set of intents, and wherein each intent-specific neural network is configured to process the features of the vehicle to generate an output for the corresponding intent.

First claim

What is claimed is:

1. A method comprising:
obtaining, by an autonomous vehicle navigating in an environment, an input characterizing one or more other vehicles in the environment;
generating, by the autonomous vehicle and from the input, features of each of the one or more other vehicles;
for each of the one or more other vehicles:
processing, by the autonomous vehicle, the features of the other vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent from a set of intents, wherein each intent from the set of intents is an action goal for the other vehicle over a first time scale, and wherein each intent-specific neural network is configured to process the features of the other vehicle to generate an output for the corresponding intent that includes:
(i) a confidence score that represents a predicted likelihood that the other vehicle will follow the intent that corresponds to the intent-specific neural network, and
(ii) a predicted trajectory that would be followed by the other vehicle in a future time period if the other vehicle follows the intent that corresponds to the intent-specific neural network, wherein the predicted trajectory is a sequence of predicted positions of the other vehicle over a second time scale that is shorter than the first time scale; and

controlling the autonomous vehicle navigating in the environment based on the outputs generated by the plurality of intent-specific neural networks for each of the one or more other vehicles.

2. The method of claim 1 , wherein controlling the autonomous vehicle navigating in the environment based on the outputs comprises:
providing the outputs generated by the plurality of intent-specific neural networks for each of the one or more other vehicles to a planning system that plans navigation of the autonomous vehicle navigating in the environment.

3. The method of claim 1 , wherein the input comprises, for each of the one or more other vehicles, an appearance embedding of the other vehicle that characterizes an appearance of the other vehicle as sensed by one or more sensors of a particular other vehicle in the environment.

4. The method of claim 1 , wherein the input comprises one or more images of the environment, and wherein generating the features of each of the one or more other vehicles comprises:
processing the one or more images of the environment using a convolutional neural network to generate a feature map that includes a respective feature vector for each of a plurality of positions in the environment; and
generating, for each of the one or more other vehicles, features based on the feature vectors in the feature map.

5. The method of claim 4 , wherein generating, for each of the one or more other vehicles, features based on the feature vectors in the feature map comprises:
obtaining a position of the other vehicle in each of the one or more images of the environment;
obtaining a plurality of cropped feature vectors, comprising a respective feature vector for each position of the other vehicle by cropping the feature map based on each position of the other vehicle; and
generating a trajectory feature map of the other vehicle by performing average-pooling operation over the plurality of cropped feature vectors.

6. The method of claim 4 , wherein generating, for each of the one or more other vehicles, features based on the feature vectors in the feature map comprises:
generating a context feature map of the other vehicle by performing average-pooling operation over the feature vectors in the feature map.

7. The method of claim 4 , wherein generating, for each of the one or more other vehicles, features based on the feature vectors in the feature map comprises:
applying a self-attention mechanism to the feature map to generate an attended feature map; and
generating the features based on the attended feature map.

8. The method of claim 1 , wherein, for each of the one or more other vehicles, the predicted trajectory comprises predicted positions of the other vehicle at each of a plurality of future time steps.

9. The method of claim 1 , wherein each intent-specific neural network is one or more fully-connected neural network layers that are configured to generate the output for the corresponding intent from the features.

10. The method of claim 1 , wherein each intent-specific neural network comprises:
one or more fully-connected neural network layers that are configured to generate the confidence score; and
one or more auto-regressive neural network layers that are configured to auto-regressively generate the predicted trajectory.

11. The method of claim 10 , wherein the one or more auto-regressive neural network layers are recurrent neural network layers.

12. The method of claim 10 , wherein, for each of the one or more other vehicles, the predicted trajectory comprises predicted positions of the other vehicle for each of a plurality of future time steps, wherein the plurality of future time steps are divided into a plurality of partitions, and wherein the auto-regressive neural network layers are configured to generate, for each partition, the predicted positions of the other vehicle for the partition conditioned on predicted positions for earlier partitions.

13. The method of claim 12 , wherein each partition includes more than one future time step.

14. The method of claim 1 , wherein each predicted trajectory is a sequence of predicted positions of the other vehicle.

15. The method of claim 1 , wherein the set of intents includes one or more of:
going straight, turning left, turning right, left lane change, right lane change, remaining stationary, or reversing.

16. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
obtaining, by an autonomous vehicle navigating in an environment, an input characterizing one or more other vehicles in the environment;
generating, by the autonomous vehicle and from the input, features of each of the one or more other vehicles;
for each of the one or more other vehicles:
processing, by the autonomous vehicle, the features of the other vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent from a set of intents, wherein each intent from the set of intents is an action goal for the other vehicle over a first time scale, and wherein each intent-specific neural network is configured to process the features of the other vehicle to generate an output for the corresponding intent that includes:
(i) a confidence score that represents a predicted likelihood that the other vehicle will follow the intent that corresponds to the intent-specific neural network, and
(ii) a predicted trajectory that would be followed by the other vehicle in a future time period if the other vehicle follows the intent that corresponds to the intent-specific neural network, wherein the predicted trajectory is a sequence of predicted positions of the other vehicle over a second time scale that is shorter than the first time scale; and

controlling the autonomous vehicle navigating in the environment based on the outputs generated by the plurality of intent-specific neural networks for each of the one or more other vehicles.

17. The system of claim 16 , wherein controlling the autonomous vehicle navigating in the environment

Assignees

• Waymo Llc

Inventors

• Gao Jiyang
• Mao Junhua
• Shen Yi
• Li Congcong
• Sun Chen

Classifications

• B60W60/00276 Primary

  for two or more other traffic participants · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/045

  Combinations of networks · CPC title