Generating trajectory labels from short-term intention and long-term result

Patent metadata

Field	Value
Publication number	US-11900224-B2
Application number	US-201916727724-A
Country	US
Kind code	B2
Filing date	Dec 26, 2019
Priority date	Dec 26, 2019
Publication date	Feb 13, 2024
Grant date	Feb 13, 2024

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating training data for training a machine learning model to perform trajectory prediction. One of the methods includes: obtaining a training input, the training input including (i) data characterizing an agent in an environment as of a first time and (ii) data characterizing a candidate trajectory of the agent in the environment for a first time period that is after the first time. A long-term label for the candidate trajectory that indicates whether the agent actually followed the candidate trajectory for the first time period is determined. A short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory is determined. A ground-truth probability for the candidate trajectory is determined. The training input is associated with the ground-truth probability for the candidate trajectory in the training data.

First claim

What is claimed is:

1. A method of generating training data for training a machine learning model to perform trajectory prediction, comprising:
obtaining a training input, the training input comprising (i) data characterizing an agent in an environment as of a first time and (ii) data characterizing a candidate trajectory of the agent in the environment for a first time period that is after the first time;
determining a long-term label for the candidate trajectory that indicates whether the agent actually followed the candidate trajectory for the first time period after the first time;
determining a short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory;
determining, based on the long-term label and the short-term label for the candidate trajectory, a ground-truth probability for the candidate trajectory, wherein the ground-truth probability for the candidate trajectory is a probability that should be assigned to the candidate trajectory by the machine learning model;
associating the training input with the ground-truth probability for the candidate trajectory in the training data; and
training the machine learning model on the training data.

2. The method of claim 1 , wherein determining, based on the long-term label and the short-term label for the candidate trajectory, a ground-truth probability for the candidate trajectory comprises:
determining that the long-term label indicates that agent actually followed the candidate trajectory after the first time;
determining that the short-term label indicates that the agent intended to follow the candidate trajectory; and
based on determining that the long-term label indicates that agent actually followed the candidate trajectory after the first time and determining that the short-term label indicates that the agent intended to follow the candidate trajectory, setting the ground-truth probability for the candidate trajectory equal to 1.

3. The method of claim 1 , wherein determining, based on the long-term label and the short-term label for the candidate trajectory, a ground-truth probability for the candidate trajectory comprises:
determining that the long-term label indicates that agent did not actually follow the candidate trajectory after the first time;
determining that the short-term label indicates that the agent did not intend to follow the candidate trajectory; and
based on determining that the long-term label indicates that agent did not actually follow the candidate trajectory after the first time and determining that the short-term label indicates that the agent did not intend to follow the candidate trajectory, setting the ground-truth probability for the candidate trajectory equal to 0.

4. The method of claim 1 , wherein determining, based on the long-term label and the short-term label for the candidate trajectory, a ground-truth probability for the candidate trajectory comprises:
determining that the long-term label indicates that agent actually followed the candidate trajectory after the first time;
determining that the short-term label indicates that the agent did not intend to follow the candidate trajectory; and
based on determining that the long-term label indicates that agent actually followed the candidate trajectory after the first time and determining that the short-term label indicates that the agent did not intend to follow the candidate trajectory, setting the ground-truth probability for the candidate trajectory equal to a first value between zero and one.

5. The method of claim 1 , wherein determining, based on the long-term label and the short-term label for the candidate trajectory, a ground-truth probability for the candidate trajectory comprises:
determining that the long-term label indicates that agent did not actually follow the candidate trajectory after the first time;
determining that the short-term label indicates that the agent intended to follow the candidate trajectory; and
based on determining that the long-term label indicates that agent did not actually follow the candidate trajectory after the first time and determining that the short-term label indicates that the agent intended to follow the candidate trajectory, setting the ground-truth probability for the candidate trajectory equal to a second value between zero and one.

6. The method of claim 1 , wherein determining, based on the long-term label and the short-term label for the candidate trajectory, a ground-truth probability for the candidate trajectory comprises:
determining that the long-term label is inconsistent with the short-term label, comprising:
determining that the long-term label indicates that agent did not actually follow the candidate trajectory after the first time and the short-term label indicates that the agent intended to follow the candidate trajectory, or
determining that the long-term label indicates that agent actually followed the candidate trajectory after the first time and determining that the short-term label indicates that the agent did not intend to follow the candidate trajectory;

obtaining context information of the environment;
based on the context information of the environment, generating an updated short-term label using one or more predetermined rules; and
based on the long-term label and the updated short-term label for the candidate trajectory, determining a ground-truth probability for the candidate trajectory.

7. The method of claim 1 , wherein determining a long-term label for the candidate trajectory that indicates whether the agent actually followed the candidate trajectory after the first time comprises:
determining whether log data that tracks movement of the agent after the first time indicates that the agent followed the candidate trajectory after the first time.

8. The method of claim 1 , wherein determining a short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory comprises:
determining whether the agent followed the candidate trajectory for an initial time period immediately after the first time, wherein the initial time period is shorter than the first time period.

9. The method of claim 1 , wherein determining a short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory comprises:
determining whether the agent had a heading that matches a heading required to follow the candidate trajectory for an initial time period immediately after the first time.

10. The method of claim 1 , wherein determining a short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory comprises:
determining, from appearance information characterizing an appearance of the agent within a first time window after the first time, whether the agent intended to follow the candidate trajectory.

11. The method of claim 10 , wherein the agent is a vehicle, and wherein the appearance information indicates whether any of one or more turn signals of the vehicle are turned on.

12. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
obtaining a training input, the training input comprising (i) data characterizing an agent in an environment as of a first time and (ii) data characterizing a candidate trajectory of the agent in the environment for a first time period that is after the first time;
determining a long-term label for the candidate trajectory t

Assignees

• Waymo Llc

Inventors

• Refaat Khaled
• Ross Stephane

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• B60W50/0097

  Predicting future conditions · CPC title

• B60W60/0027

  using trajectory prediction for other traffic participants · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title