Training machine learning model based on training instances with: training instance input based on autonomous vehicle sensor data, and training instance output based on additional vehicle sensor data

Patent metadata

Field	Value
Publication number	US-10676085-B2
Application number	US-201816173669-A
Country	US
Kind code	B2
Filing date	Oct 29, 2018
Priority date	Apr 11, 2018
Publication date	Jun 9, 2020
Grant date	Jun 9, 2020

Abstract

Various implementations described herein generate training instances that each include corresponding training instance input that is based on corresponding sensor data of a corresponding autonomous vehicle, and that include corresponding training instance output that is based on corresponding sensor data of a corresponding additional vehicle, where the corresponding additional vehicle is captured at least in part by the corresponding sensor data of the corresponding autonomous vehicle. Various implementations train a machine learning model based on such training instances. Once trained, the machine learning model can enable processing, using the machine learning model, of sensor data from a given autonomous vehicle to predict one or more properties of a given additional vehicle that is captured at least in part by the sensor data.

First claim

What is claimed is:

1. A method of training a machine learning model to be used in autonomous control of at least one autonomous vehicle, the method comprising:
generating a plurality of training instances, wherein generating each of the training instances includes:
generating training instance input of the training instance based on a corresponding instance of vision data from a vision component of a corresponding autonomous vehicle; and
generating a supervised training instance output of the training instance based on a corresponding instance of additional vehicle data, wherein the corresponding instance of additional vehicle data is based on one or more sensors of a corresponding additional vehicle that is captured by the corresponding instance of vision data, and wherein the corresponding instance of additional vehicle data indicates a corresponding current state of at least one dynamic property of the corresponding additional vehicle;

wherein the corresponding instance of additional vehicle data is used in generating the supervised training instance output based on determining that the corresponding instance of additional vehicle data temporally corresponds to the corresponding instance of vision data;
training the machine learning model based on the plurality of training instances; and
providing the trained machine learning model for use in control of the at least one autonomous vehicle.

2. The method of claim 1 , wherein the corresponding instance of vision data is light Detection and Ranging (LIDAR) data and the vision component is a LIDAR component.

3. The method of claim 2 , wherein the LIDAR component is a phase coherent LIDAR component and the LIDAR data comprises a group of LIDAR data points of a sensing cycle of the phase coherent LIDAR component, each of the LIDAR data points of the group indicating a corresponding range and a corresponding velocity for a corresponding point in the environment, and each being generated based on a corresponding sensing event of the sensing cycle.

4. The method of claim 1 , wherein the supervised training instance output of the training instance further includes an indication of a bounding area that bounds a portion of the vision data that captures the corresponding additional vehicle.

5. The method of claim 1 , wherein training the machine learning model comprises:
processing, using the machine learning model, the training instance inputs of the training instances to generate predicted outputs;
generating losses based on the predicted outputs and the supervised training instance outputs of the training instances; and
updating the machine learning model based on the generated losses.

6. The method of claim 1 , wherein the instance of additional vehicle data comprises one or more yaw parameters and the one or more sensors of the corresponding additional vehicle comprise at least a yaw rate sensor.

7. The method of claim 6 , wherein the additional vehicle data is determined based on monitoring of a controller area network of the corresponding additional vehicle.

8. The method of claim 1 , wherein the instance of additional vehicle data comprises one or both of velocity and acceleration of the corresponding additional vehicle.

9. The method of claim 1 , wherein the instance of additional vehicle data comprises yaw parameters and one or both of velocity and acceleration.

10. The method of claim 1 , further comprising:
processing, by at least one processor of a given autonomous vehicle of the at least one autonomous vehicle, given vision data using the trained machine learning model, the given vision data captured by a given vision component of the given autonomous vehicle;
generating, based on the processing, a predicted state of a given additional vehicle captured by the given vision data; and
controlling the given autonomous vehicle based on the predicted state.

11. The method of claim 10 , wherein the given vision data is a subgroup of vision data captured during a sensing cycle of the vision component, and further comprising:
generating the subgroup of vision data based on determining the subgroup corresponds to the given additional vehicle.

12. The method of claim 11 , wherein determining the subgroup corresponds to the given additional vehicle comprises:
processing the vision data using an additional object detection and classification model; and
determining the subgroup corresponds to the given additional vehicle based on output generated based on processing the vision data using the additional object detection and classification model.

13. A method of controlling an autonomous vehicle using a trained machine learning model, the method implemented by one or more processors of the autonomous vehicle, and the method comprising:
processing given vision data using the trained machine learning model, the given vision data captured by a vision component of the autonomous vehicle;
generating, based on the processing, a predicted state of at least one dynamic property of a given additional vehicle captured by the given vision data; and
controlling the autonomous vehicle based on the predicted state;
wherein the trained machine learning model is trained based on a plurality of training instances that each includes:
training instance input that is based on a corresponding instance of vision data from a vision component of a corresponding autonomous vehicle; and
supervised training instance output that is based on a corresponding instance of additional vehicle data, wherein the corresponding instance of additional vehicle data is based on one or more sensors of a corresponding additional vehicle that is captured by the corresponding instance of vision data.

14. The method of claim 13 , wherein the given vision data is a subgroup of vision data captured during a sensing cycle of the vision component, and further comprising:
generating the subgroup of vision data based on determining the subgroup corresponds to the given additional vehicle.

15. The method of claim 13 , wherein determining the subgroup corresponds to the given additional vehicle comprises:
processing the vision data using an additional object detection and classification model; and
determining the subgroup corresponds to the given additional vehicle based on output generated based on processing the vision data using the additional object detection and classification model.

16. The method of claim 13 , wherein controlling the autonomous vehicle comprises controlling speed and/or direction of the autonomous vehicle.

17. The method of claim 13 , wherein the at least one property comprises one or both of a yaw rate and a yaw direction of the additional vehicle.

18. The method of claim 13 , wherein the at least one property comprises one or both of velocity and acceleration of the additional vehicle.

19. The method of claim 13 , wherein the at least one property comprises a yaw parameter and one or both of velocity and acceleration of the additional vehicle.

Assignees

• Aurora Innovation Inc

Inventors

• Smith Warren
• Eade Ethan
• Anderson Sterling J
• Bagnell James Andrew
• Nabbe Bartholomeus C
• Urmson Christopher Paul

Classifications

• B60W2554/803

  Relative lateral speed · CPC title

• B60W2554/4041

  Position · CPC title

• G06N20/00

  Machine learning · CPC title

• B60W30/0956 Primary

  the prediction being responsive to traffic or environmental parameters · CPC title

• G05D1/024

  in combination with a laser (lasers per se H01S) · CPC title