Sensor aggregation framework for autonomous driving vehicles

Patent metadata

Field	Value
Publication number	US-12017663-B2
Application number	US-202117205725-A
Country	US
Kind code	B2
Filing date	Mar 18, 2021
Priority date	Apr 12, 2018
Publication date	Jun 25, 2024
Grant date	Jun 25, 2024

Abstract

A sensor aggregation framework for autonomous driving vehicles is disclosed. In one embodiment, sensor data is collected from one or more sensors mounted on an autonomous driving vehicle (ADV) while the ADV is moving within a region of interest (ROI) that includes a number of obstacles. The sensor data includes obstacle information of the obstacles and vehicle data of the ADV. Each of the vehicle data is timestamped with a current time at which the vehicle data is captured to generate a number of timestamps that correspond to the vehicle data. The obstacle information, the vehicle data, and the corresponding timestamps are aggregated into training data. The training data is used to train a set of parameters that is subsequently utilized to predict at least in part future obstacle behaviors and vehicle movement of the ADV.

First claim

What is claimed is:

1. A computer-implemented method of training a set of parameters for subsequent use in controlling an autonomous driving vehicle, comprising:
collecting sensor data from one or more sensors mounted on an autonomous driving vehicle (ADV) while the ADV is moving within a region of interest (ROI) that includes a plurality of obstacles, the sensor data including obstacle information of the obstacles and vehicle data of the ADV, each of the vehicle data including driving commands issued and responses of the ADV at a specific point in time, wherein the driving commands issued include a throttle, brake, or steering command of the ADV, and the responses of the ADV include a speed, acceleration, deceleration, or direction of the ADV;
timestamping each of the vehicle data with a current time at which the vehicle data is captured to generate a plurality of timestamps that correspond to the vehicle data, the plurality of timestamps being mapped to the specific point in time of the driving commands issued and the responses of the ADV; and
aggregating the obstacle information, the vehicle data, and the corresponding timestamps into training data to align timing of the obstacle information with the plurality of timestamps of the throttle, the brake, or the steering command of the driving commands issued and the speed, the acceleration, the deceleration, or the direction of the responses of the ADV, wherein the training data is used to train a set of parameters that is subsequently utilized to predict at least in part future obstacle behaviors and vehicle movement of the ADV.

2. The method of claim 1 , further comprising:
prior to aggregating the obstacle information, the vehicle data, and the corresponding timestamps into the training data, extracting the obstacle information and the vehicle data from the sensor data, and
subsequent to aggregating the obstacle information, the vehicle data, and the corresponding timestamps into the training data, training the set of parameters using the training data to learn obstacle behaviors of the obstacles and current vehicle movement of the ADV.

3. The method of claim 2 , wherein training the set of parameters comprises invoking a machine learning model to continuously learn the obstacle information, the vehicle data and the corresponding timestamps.

4. The method of claim 1 , wherein aggregating the obstacle information, the vehicle data, and the corresponding timestamps into the training data comprises:
appending each of the vehicle data and its corresponding timestamp to each other in a pairwise manner to form a plurality of pairs of vehicle data and corresponding timestamp,
appending the vehicle data and corresponding timestamp pairs to each other, and
appending the appended vehicle data and corresponding timestamp pairs to the obstacle information.

5. The method of claim 1 , wherein the training data includes a plurality of appended vehicle data and corresponding timestamp pairs appended to the obstacle information, wherein each of the obstacle information is appended to each other.

6. The method of claim 1 , wherein the obstacle information includes positions and reflectivity of the obstacles.

7. The method of claim 1 , wherein collecting sensor data from one or more sensors mounted on the ADV is performed within a specific time frame.

8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
collecting sensor data from one or more sensors mounted on an autonomous driving vehicle (ADV) while the ADV is moving within a region of interest (ROI) that includes a plurality of obstacles, the sensor data including obstacle information of the obstacles and vehicle data of the ADV, each of the vehicle data including driving commands issued and responses of the ADV at a specific point in time, wherein the driving commands issued include a throttle, brake, or steering command of the ADV, and the responses of the ADV include a speed, acceleration, deceleration, or direction of the ADV;
timestamping each of the vehicle data with a current time at which the vehicle data is captured to generate a plurality of timestamps that correspond to the vehicle data, the plurality of timestamps being mapped to the specific point in time of the driving commands issued and the responses of the ADV; and
aggregating the obstacle information, the vehicle data, and the corresponding timestamps into training data to align timing of the obstacle information with the plurality of timestamps of the throttle, the brake, or the steering command of the driving commands issued and the speed, the acceleration, the deceleration, or the direction of the responses of the ADV, wherein the training data is used to train a set of parameters that is subsequently utilized to predict at least in part future obstacle behaviors and vehicle movement of the ADV.

9. The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise:
prior to aggregating the obstacle information, the vehicle data, and the corresponding timestamps into the training data, extracting the obstacle information and the vehicle data from the sensor data, and
subsequent to aggregating the obstacle information, the vehicle data, and the corresponding timestamps into the training data, training the set of parameters using the training data to learn obstacle behaviors of the obstacles and current vehicle movement of the ADV.

10. The non-transitory machine-readable medium of claim 9 , wherein training the set of parameters comprises invoking a machine learning model to continuously learn the obstacle information, the vehicle data and the corresponding timestamps.

11. The non-transitory machine-readable medium of claim 8 , wherein aggregating the obstacle information, the vehicle data, and the corresponding timestamps into the training data comprises:
appending each of the vehicle data and its corresponding timestamp to each other in a pairwise manner to form a plurality of pairs of vehicle data and corresponding timestamp,
appending the vehicle data and corresponding timestamp pairs to each other, and
appending the appended vehicle data and corresponding timestamp pairs to the obstacle information.

12. The non-transitory machine-readable medium of claim 8 , wherein the training data includes a plurality of appended vehicle data and corresponding timestamp pairs appended to the obstacle information, wherein each of the obstacle information is appended to each other.

13. The non-transitory machine-readable medium of claim 8 , wherein the obstacle information includes positions and reflectivity of the obstacles.

14. The non-transitory machine-readable medium of claim 8 , wherein collecting sensor data from one or more sensors mounted on the ADV is performed within a specific time frame.

15. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
collecting sensor data from one or more sensors mounted on an autonomous driving vehicle (ADV) while the ADV is moving within a region of interest (ROI) that includes a plurality of obstacles, the sensor data including obstacle information of the obstacles and vehicle data of the ADV, each of the vehicle data including driving commands issued and responses of the ADV at a specific point in time, wherein the driving commands issued include a throttle, brake, or steering command of the ADV,

Assignees

• Baidu Usa Llc

Inventors

• Zhang Liangliang
• Li Dong
• Hu Jiangtao
• Tao Jiaming
• Jiang Yifei

Classifications

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W60/00274

  considering possible movement changes · CPC title

• B60W60/00276

  for two or more other traffic participants · CPC title

• B60W50/0097 Primary

  Predicting future conditions · CPC title

• B60W60/0027 Primary

  using trajectory prediction for other traffic participants · CPC title