Trajectory representation in behavior prediction systems

Patent metadata

Field	Value
Publication number	US-11003189-B2
Application number	US-202016922798-A
Country	US
Kind code	B2
Filing date	Jul 7, 2020
Priority date	Nov 20, 2018
Publication date	May 11, 2021
Grant date	May 11, 2021

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a representation of a trajectory of a target agent in an environment. In one aspect, the representation of the trajectory of the target agent in the environment is a concatenation of a plurality of channels, where each channel is represented as a two-dimensional array of data values. Each position in each channel corresponds to a respective spatial position in the environment, and corresponding positions in different channels correspond to the same spatial position in the environment. The channels include a time channel and a respective motion channel corresponding to each motion parameter in a predetermined set of motion parameters.

First claim

What is claimed is:

1. A method performed by one or more data processing apparatus, the method comprising:
obtaining historical data characterizing a trajectory of a target agent in an environment, the historical data comprising, for each of a plurality of previous time points, data defining a spatial position in the environment occupied by the target agent at the previous time point;
generating a representation of the trajectory of the target agent in the environment, wherein:
the representation of the trajectory of the target agent in the environment comprises a time channel;
the time channel is represented as a two-dimensional array of data values;
each position in the time channel corresponds to a respective spatial position in the environment; and
for each particular spatial position in the environment occupied by the target agent at a particular previous time point:
the position in the time channel which corresponds to the particular spatial position defines the particular previous time point; and

processing an input which comprises the representation of the trajectory of the target agent in the environment using a convolutional neural network to generate data characterizing a future trajectory of the target agent in the environment after a current time point.

2. The method of claim 1 , wherein the historical data further comprises, for each of the plurality of previous time points, respective values of each motion parameter in a predetermined set of motion parameters, wherein the value of each motion parameter characterizes a respective feature of a motion of the target agent at the previous time point.

3. The method of claim 2 , wherein:
the representation of the trajectory of the target agent in the environment comprises a concatenation of a plurality of channels, including: (i) the time channel, and (ii) a respective motion channel corresponding to each motion parameter in the predetermined set of motion parameters;
each motion channel is represented as a two-dimensional array of data values;
each position in each motion channel corresponds to a respective spatial position in the environment; and
corresponding positions in different channels of the plurality of channels correspond to the same spatial position in the environment.

4. The method of claim 3 , wherein for each particular spatial position in the environment occupied by the target agent at a particular previous time point:
for each motion channel, the position in the motion channel which corresponds to the particular spatial position defines the value of the motion parameter corresponding to the motion channel at the particular previous time point.

5. The method of claim 4 , wherein the set of motion parameters comprises:
a velocity motion parameter that characterizes a velocity of the target agent in the environment;
an acceleration motion parameter that characterizes an acceleration of the target agent in the environment; and
a heading motion parameter that characterizes a heading of the target agent in the environment.

6. The method of claim 1 , further comprising:
processing the data characterizing the future trajectory of the target agent in the environment after the current time point using a planning system of a vehicle to generate a control output; and
autonomously modifying operations of the vehicle based on the control output.

7. The method of claim 1 , wherein processing an input which comprises the representation of the trajectory of the target agent in the environment using a convolutional neural network to generate data characterizing a future trajectory of the target agent in the environment after the current time point comprises:
processing an input which comprises the representation of the trajectory of the target agent in the environment to generate data defining, for each of a plurality of spatial locations in the environment, a respective probability that the target agent will occupy the spatial location at a specified time point after the current time point.

8. The method of claim 1 , wherein processing an input which comprises the representation of the trajectory of the target agent in the environment using a convolutional neural network to generate data characterizing a future trajectory of the target agent in the environment after the current time point comprises:
processing an input comprising data defining a candidate future trajectory of the target agent in the environment after the current time point in addition to the representation of the trajectory of the target agent in the environment to generate data defining a probability that the future trajectory of the target agent is the candidate future trajectory of the target agent.

9. The method of claim 1 , wherein processing an input which comprises the representation of the trajectory of the target agent in the environment using a convolutional neural network to generate data characterizing a future trajectory of the target agent in the environment after the current time point comprises:
generating data defining respective probabilities that the target agent will make each of a predetermined number of possible driving decisions.

10. The method of claim 1 , wherein the time channel represents a top-down perspective of the environment.

11. The method of claim 1 , wherein processing an input which comprises the representation of the trajectory of the target agent in the environment using a convolutional neural network comprises:
processing an input comprising one or more of: (i) a road-graph channel representing a known geometry of the environment, (ii) a target agent localization channel which represents a spatial position of the target agent in the environment at the current time point by a target agent bounding box, and (iii) an auxiliary localization channel which represents respective spatial positions of one or more other agents in the environment at the current time point by respective other agent bounding boxes.

12. The method of claim 2 , further comprising:
generating a joint representation of trajectories of a plurality of other agents in the environment; and
processing an input which comprises the joint representation of the trajectories of the plurality of other agents in the environment, in addition to the representation of the trajectory of the target agent in the environment, using the convolutional neural network to generate data characterizing the future trajectory of the target agent in the environment after the current time point.

13. The method of claim 12 , wherein:
the joint representation of the trajectories of the plurality of other agents in the environment comprises an auxiliary time channel and a respective auxiliary motion channel corresponding to each motion parameter in the predetermined set of motion parameters;
each channel is represented as a two-dimensional array of data values and each data value in each channel corresponds to a respective spatial position in the environment; and
for each particular spatial position in the environment occupied by a particular other agent of the plurality of other agents at a particular previous time point:
the data value in the auxiliary time channel which corresponds to the particular spatial position defines the particular previous time point; and
for each auxiliary motion channel, the data value in the auxiliary motion channel which corresponds to the particular spatial position defines a value of the motion parameter corresponding to the auxiliary motion channel which characterizes a respective feature of a motion of the particular other agent at the particular previous time point.

Assignees

• Waymo Llc

Inventors

• Refaat Khaled
• Ross Stephane

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G05D1/0221 Primary

  involving a learning process · CPC title

• G06V10/82 Primary

  using neural networks · CPC title