Future object trajectory predictions for autonomous machine applications

Patent metadata

Field	Value
Publication number	US-11514293-B2
Application number	US-201916564978-A
Country	US
Kind code	B2
Filing date	Sep 9, 2019
Priority date	Sep 11, 2018
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

In various examples, historical trajectory information of objects in an environment may be tracked by an ego-vehicle and encoded into a state feature. The encoded state features for each of the objects observed by the ego-vehicle may be used—e.g., by a bi-directional long short-term memory (LSTM) network—to encode a spatial feature. The encoded spatial feature and the encoded state feature for an object may be used to predict lateral and/or longitudinal maneuvers for the object, and the combination of this information may be used to determine future locations of the object. The future locations may be used by the ego-vehicle to determine a path through the environment, or may be used by a simulation system to control virtual objects—according to trajectories determined from the future locations—through a simulation environment.

First claim

What is claimed is:

1. A method comprising:
positioning, within a spatial arrangement, one or more encoded state features, each encoded state feature of the one or more encoded state features corresponding to a respective object in an environment, each encoded state feature being positioned at a location within the spatial arrangement that corresponds to a relative location of the respective object in the environment with respect to an ego-vehicle;
computing, using a long short-term memory (LSTM) network, a first encoded spatial feature by inputting, to the LSTM network, first data representative of the spatial arrangement traversed in a first direction;
computing, using the LSTM network or another LSTM network, a second encoded spatial feature by inputting, to the LSTM network or the another LSTM network, second data representative of the spatial arrangement traversed in a second direction different from the first direction; and
for each respective object, computing confidence values corresponding to a likelihood of occurrence of a plurality of object maneuvers based at least in part on the encoded state feature corresponding to the respective object, the first encoded spatial feature, and the second encoded spatial feature.

2. The method of claim 1 , wherein each encoded state feature is computed using a machine learning model and based at least in part on a plurality of prior locations within the environment of the respective object corresponding to the encoded state feature.

3. The method of claim 1 , wherein the plurality of object maneuvers include lateral maneuvers selected from changing to a right adjacent lane, changing to a left adjacent lane, and staying in a current lane, and the plurality of object maneuvers further include longitudinal maneuvers selected from accelerating, decelerating, or maintaining a current speed.

4. The method of claim 1 , further comprising:
based at least in part on the confidence values, determining one or more object maneuvers from the plurality of object maneuvers;
for each respective object, computing a density function corresponding to a space of future locations for the respective object based at least in part on the encoded state feature corresponding to the respective object, the first encoded spatial feature, the second encoded spatial feature, and the one or more object maneuvers.

5. The method of claim 4 , wherein the density function represents a region of locations within the space of one or more future locations, the region of locations computed using at least one of: a standard deviation of the one or more future locations, a mean of the one or more future locations, or a covariance of the one or more future locations.

6. The method of claim 4 , further comprising:
inputting, to a machine learning model, third data representative of a future location within the environment, the machine learning model for computing the one or more encoded state features; and
computing, using the machine learning model, an updated encoded state feature for the respective object using the third data.

7. The method of claim 6 , wherein the updated encoded state feature is used to compute another future location after the future location for the respective object.

8. The method of claim 1 , wherein the environment is a simulation environment, and at least an object maneuver from the plurality of object maneuvers is used to determine a trajectory for the respective object within the simulation environment.

9. The method of claim 1 , wherein each encoded state feature of the one or more encoded state features is computed using a respective instantiation of an LSTM network.

10. A method comprising:
inputting, at a first time instance and to a long short-term memory (LSTM) network, first data representative of one or more past locations and a current location of an object in an environment;
computing, using the LSTM network and based at least in part on the first data, a first encoded state feature corresponding to the object at the first time instance;
computing, at the first time instance, an encoded spatial feature using the first encoded state feature and one or more additional encoded state features corresponding to one or more additional objects in the environment;
predicting a future location of the object in the environment using the encoded spatial feature and the first encoded state feature;
inputting, at a second time instance after the first time instance and to the LSTM network, second data representative of one or more updated past locations, a new current location, and the future location of the object in the environment; and
computing, using the LSTM network and based at least in part on the second data, a second encoded state feature corresponding to the object at the second time instance.

11. The method of claim 10 , wherein the one or more past locations and the current location are represented as values relative to an origin point of an ego-vehicle in the environment.

12. The method of claim 10 , wherein the computing the encoded spatial feature includes:
populating a spatial arrangement to include the first encoded state feature and the one or more additional encoded state features, the populating including positioning the first encoded state feature and the one or more additional encoded state features at spatial arrangement locations within the spatial arrangement, the spatial arrangement locations corresponding to relative locations of the object and the one or more additional objects in the environment with respect to an ego-vehicle; and
inputting, to a machine learning model, third data representative of the spatial arrangement.

13. The method of claim 10 , wherein the first encoded state feature, the second encoded state feature, and the encoded spatial feature are computed based at least in part on sensor data generated by one or more sensors of an ego-vehicle.

14. The method of claim 10 , wherein the encoded spatial feature is computed using a bi-directional LSTM network.

15. The method of claim 10 , further comprising:
determining a predicted maneuver from a plurality of maneuvers based at least in part on the first encoded state feature and the encoded spatial feature,
wherein the predicting the future location of the object is further based at least in part on the predicted maneuver.

16. The method of claim 15 , wherein the predicted maneuver includes a longitudinal component and a lateral component.

17. A system comprising:
one or more sensors to generate sensor data of an environment around a vehicle;
a computing device including one or more processing devices and one or more memory devices communicatively coupled to the one or more processing devices storing programmed instructions thereon, which when executed by the one or more processing devices causes instantiation of:
a temporal encoder to generate one or more encoded state features each corresponding to an object in an environment;
a spatial encoder to generate an encoded spatial feature from a spatial arrangement, the spatial arrangement including the one or more encoded state features;
a maneuver predictor to determine, based at least in part on the encoded spatial feature and an encoded state feature of the one or more encoded state features, a likelihood of one or more predicted maneuvers of a respective object corresponding to the encoded state feature; and
a trajectory decoder to determine one or more future locations of the respective object based at least in part on the encoded state feature, the encoded spatial feature, and the one

Assignees

• Nvidia Corp

Inventors

• Villegas Ruben
• Troccoli Alejandro
• Frosio Iuri
• Tyree Stephen
• Byeon Wonmin
• Kautz Jan

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/044 Primary

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title