Pedestrian crossing intent yielding

Patent metadata

Field	Value
Publication number	US-12344273-B2
Application number	US-202217902670-A
Country	US
Kind code	B2
Filing date	Sep 2, 2022
Priority date	Sep 2, 2021
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium that determine yield behavior for an autonomous vehicle, and can include identifying an agent that is in a vicinity of an autonomous vehicle navigating through a scene at a current time point. Scene features can be obtained and can include features of (i) the agent and (ii) the autonomous vehicle. An input that can include the scene features can be processed using a first machine learning model that is configured to generate (i) a crossing intent prediction that includes a crossing intent score that represents a likelihood that the agent intends to cross a roadway in a future time window after the current time, and (ii) a crossing action prediction that includes a crossing action score that represents a likelihood that the agent will cross the roadway in the future time window after the current time.

First claim

What is claimed is:

1. A method performed by one or more computers, the method comprising:
identifying an agent that is in a vicinity of an autonomous vehicle navigating through an scene at a current time point;
obtaining scene features of the scene, the scene features comprising features of (i) the agent and (ii) the autonomous vehicle;
processing the scene features to generate a combined representation of the scene;
processing the combined representation using a first machine learning model that is configured to generate a crossing signal that comprises both (i) a crossing intent prediction for the agent that comprises a crossing intent score that represents a likelihood that the agent intends to cross a roadway in a future time window after the current time point, and (ii) a crossing action prediction for the agent that comprises a crossing action score that represents a likelihood that the agent will cross the roadway in the future time window after the current time point;
determining, from the crossing intent score, that the agent has an intent to cross the roadway in the future time window after the current time point;
determining, from the crossing action score, that the agent has a predicted crossing action of yielding for the future time window after the current time point;
determining whether the autonomous vehicle should yield to the agent based on (i) determining that the agent has an intent to cross the roadway in the future time window after the current time point and (ii) determining that the agent has a predicted crossing action of yielding for the future time window after the current time point; and
controlling the autonomous vehicle based on the determination.

2. The method of claim 1 , wherein controlling the autonomous vehicle based on the determination further comprises:
determining, from at least (i) determining that the agent has an intent to cross the roadway in the future time window after the current time point and (ii) determining that the agent has a predicted crossing action of yielding for the future time window after the current time point, a future trajectory for the autonomous vehicle after the current time point.

3. The method of claim 1 , wherein processing the scene features comprises:
processing the scene features using an encoder neural network to generate an encoded representation of the scene features;
processing the encoded representation of the scene features using one or more intent prediction neural network layers to generate the crossing intent prediction; and
processing the encoded representation of the scene features using one or more crossing action prediction neural network layers to generate the crossing action prediction.

4. The method of claim 3 , wherein the scene features include a plurality of different feature categories, and wherein processing the scene features using the encoder neural network to generate the encoded representation comprises:
for each feature category, processing the features of the feature category using an encoder subnetwork corresponding to the feature category to generate an encoded representation for the feature category; and
combining the encoded representations for the feature category to generate the encoded representation of the scene features.

5. The method of claim 1 where the first machine learning model is further configured to generate a cross-in-front prediction that comprises a cross-in-front score that represents a likelihood that agent intends to cross in front of the autonomous vehicle in a future time window after the current time point.

6. The method of claim 1 where the scene features comprise at least one of sensor data, roadgraph information and object track data.

7. The method of claim 1 where the machine learning model executes in a single forward pass.

8. The method of claim 1 , wherein determining, from the crossing intent score, that the agent has an intent to cross the roadway in the future time window after current time point comprises determining that the crossing intent score satisfies a first threshold, and wherein determining, from the crossing action score, that the agent has a predicted crossing action of yielding for the future time window after the current time point comprises determining that the crossing action score does not satisfy a second threshold.

9. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
identifying an agent that is in a vicinity of an autonomous vehicle navigating through an scene at a current time point;
obtaining scene features of the scene, the scene features comprising features of (i) the agent and (ii) the autonomous vehicle;
processing the scene features to generate a combined representation of the scene;
processing the combined representation using a first machine learning model that is configured to generate a crossing signal that comprises both (i) a crossing intent prediction for the agent that comprises a crossing intent score that represents a likelihood that the agent intends to cross a roadway in a future time window after the current time point, and (ii) a crossing action prediction for the agent that comprises a crossing action score that represents a likelihood that the agent will cross the roadway in the future time window after the current time point;
determining, from the crossing intent score, that the agent has an intent to cross the roadway in the future time window after the current time point;
determining, from the crossing action score, that the agent has a predicted crossing action of yielding for the future time window after the current time point;
determining whether the autonomous vehicle should yield to the agent based on (i) determining that the agent has an intent to cross the roadway in the future time window after the current time point and (ii) determining that the agent has a predicted crossing action of yielding for the future time window after the current time point; and
controlling the autonomous vehicle based on the determination.

10. The system of claim 9 , wherein controlling the autonomous vehicle based on the determination further comprises:
determining, from at least (i) determining that the agent has an intent to cross the roadway in the future time window after the current time point and (ii) determining that the agent has a predicted crossing action of yielding for the future time window after the current time point, a future trajectory for the autonomous vehicle after the current time point.

11. The system of claim 9 , wherein processing the scene features comprises:
processing the scene features using an encoder neural network to generate an encoded representation of the scene features;
processing the encoded representation of the scene features using one or more intent prediction neural network layers to generate the crossing intent prediction; and
processing the encoded representation of the scene features using one or more crossing action prediction neural network layers to generate the crossing action prediction.

12. The system of claim 11 , wherein the scene features include a plurality of different feature categories, and wherein processing the scene features using the encoder neural network to generate the encoded representation comprises:
for each feature category, processing the features of the feature category using an encoder subnetwork corresponding to the feature category to generate an encoded representation for the feature category; and
combining the encoded representations for the feature ca

Assignees

• Waymo Llc

Inventors

• Shi Xinwei
• Mao Junhua
• Refaat Khaled
• Lan Tian
• Kang Jeonhyung
• Zhang Zhishuai
• Stroud Jonathan Chandler

Classifications

• B60W50/0097

  Predicting future conditions · CPC title

• B60W2554/4041

  Position · CPC title

• B60W2554/4029

  Pedestrians · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• B60W2554/4045

  Intention, e.g. lane change or imminent movement · CPC title