Methods and systems for predicting dynamic object behavior

Patent metadata

Field	Value
Publication number	US-12118461-B2
Application number	US-202017097840-A
Country	US
Kind code	B2
Filing date	Nov 13, 2020
Priority date	Nov 13, 2020
Publication date	Oct 15, 2024
Grant date	Oct 15, 2024

Abstract

Methods and systems for predicting behavior of a dynamic object of interest in an environment of a vehicle are described. Time series feature data are received, representing features of objects in the environment, including a dynamic object of interest. The feature data are categorized into one of a plurality of defined object categories. Each categorized set of data is encoded into a respective categorical representation that represents temporal change of features within the respective defined object category. The categorical representations are combined into a single shared representation. A categorical interaction representation is generated based on the single shared representation that represents contributions of temporal change in each defined object category to a final time step of the shared representation. The categorical interaction representation together with data representing dynamics of the objects in the environment and data representing a state of the vehicle are used to generate predicted data representing a predicted future behavior of the dynamic object of interest.

First claim

The invention claimed is:

1. A method for predicting behavior of a dynamic object of interest in an environment of a vehicle, the method comprising:
receiving a plurality of time series of feature data, each time series of feature data representing a respective feature of a plurality of objects in the environment over a plurality of time steps, the plurality of objects including a dynamic object of interest;
categorizing each time series of feature data into one of a plurality of defined object categories, each defined object category being defined by one or more shared characteristics of objects belonging to that defined object category, to obtain a categorized set of data for each defined object category, each categorized set of data containing one or more time series of feature data representing one or more objects belonging to the respective defined object category;
for each given categorized set of data belonging to a given defined object category, encoding, using a spatiotemporal encoder, the one or more time series of feature data belonging to the given categorized set of data into a single categorical representation, the single categorical representation being a time series of feature vectors representing temporal change of features within the given defined object category over the plurality of time steps, wherein a respective plurality of categorical representations is obtained for the plurality of defined object categories;
combining the plurality of categorical representations into a single shared representation that is a single time series of tensors that represents spatiotemporal changes in each of the plurality of defined object categories over the plurality of time steps;
generate a categorical interaction representation based on the single shared representation, the categorical interaction representation being a weighted representation of the single shared representation representing contributions of temporal change in each defined object category to a final time step of the single shared representation; and
generate predicted data representing a predicted future behavior of the dynamic object of interest based on the categorical interaction representation and data representing dynamics of the plurality of objects and data representing a state of the vehicle.

2. The method of claim 1 , wherein combining the categorical representations comprises concatenating the categorical representations according to time step, to generate the single shared representation.

3. The method of claim 1 , wherein encoding each categorized set of data comprises:
for the given categorized set of data belonging to the given defined object category, providing the one or more time series of feature data to a trained neural network to generate the time series of feature vectors as the single categorical representation for the given object category.

4. The method of claim 3 , wherein the trained neural network is a recursive neural network, a convolutional neural network, or a combined recursive and convolutional neural network.

5. The method of claim 1 , wherein at least one defined object category is specific to the dynamic object of interest.

6. The method of claim 1 , further comprising:
receiving a time series of sensor data generated by a sensor; and
preprocessing the time series of sensor data into one time series of feature data, the one time series of feature data being included in the received plurality of time series of feature data.

7. The method of claim 1 , further comprising:
providing the predicted data representing the predicted future behavior of the dynamic object of interest to a motion planning subsystem of the vehicle, to generate a planned path for the vehicle.

8. A computing system for predicting behavior of a dynamic object of interest in an environment of a vehicle, the computing system comprising:
a processor system configured to execute instructions to cause an object behavior prediction subsystem of the computing system to:
receive a plurality of time series of feature data, each time series of feature data representing a respective feature of a plurality of objects in the environment over a plurality of time steps, the plurality of objects including a dynamic object of interest;
categorize each time series of feature data into one of a plurality of defined object categories, each defined object category being defined by one or more shared characteristics of objects belonging to that defined object category, to obtain a categorized set of data for each defined object category, each categorized set of data containing one or more time series of feature data representing one or more objects belonging to the respective defined object category;
for each given categorized set of data belonging to a given defined object category, encode, using a spatiotemporal encoder, the one or more time series of feature data belonging to the given categorized set of data into a single categorical representation, the single categorical representation being a time series of feature vectors representing temporal change of features within the given defined object category over the plurality of time steps, wherein a respective plurality of categorical representations is obtained for the plurality of defined object categories;
combine the plurality of categorical representations into a single shared representation that is a single time series of tensors that represents spatiotemporal changes in each of the plurality of defined object categories over the plurality of time steps;
generate a categorical interaction representation based on the single shared representation, the categorical interaction representation being a weighted vector representing contributions of temporal change in each defined object category to a final time step of the shared representation; and
generate predicted data based at least on the categorical interaction representation, data representing dynamics of the objects, and data representing a state of the vehicle, the predicted data representing a predicted future behavior of the dynamic object of interest.

9. The computing system of claim 8 , wherein the processing system is configured to execute instructions to combine the categorical representations by concatenating the categorical representations according to time step, to generate the single shared representation.

10. The computing system of claim 8 , wherein the processing system is configured to execute instructions to encode each categorized set of data by:
for the given categorized set of data belonging to the given defined object category, providing the one or more time series of feature data to a trained neural network to generate the time series of feature vectors as the single categorical representation for the given object category.

11. The computing system of claim 10 , wherein the trained neural network is a recursive neural network, a convolutional neural network, or a combined recursive and convolutional neural network.

12. The computing system of claim 8 , wherein at least one defined object category is specific to the dynamic object of interest.

13. The computing system of claim 8 , wherein the processing system is configured to execute instructions to cause the computing system to:
receive a time series of sensor data generated by a sensor; and
preprocessing the time series of sensor data into one time series of feature data, the one time series of feature data being included in the received plurality of time series of feature data.

14. The computing system of claim 8 , wherein the vehicle is an autonomous vehicle, wherein the computing

Assignees

• Rasouli Amir
• Rohani Mohsen
• Huawei Tech Co Ltd

Inventors

• Rasouli Amir
• Rohani Mohsen

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title