Training machine learning model based on training instances with: training instance input based on autonomous vehicle sensor data, and training instance output based on additional vehicle sensor data

Patent metadata

Field	Value
Publication number	US-11358601-B2
Application number	US-202016869438-A
Country	US
Kind code	B2
Filing date	May 7, 2020
Priority date	Apr 11, 2018
Publication date	Jun 14, 2022
Grant date	Jun 14, 2022

Abstract

Various implementations described herein generate training instances that each include corresponding training instance input that is based on corresponding sensor data of a corresponding autonomous vehicle, and that include corresponding training instance output that is based on corresponding sensor data of a corresponding additional vehicle, where the corresponding additional vehicle is captured at least in part by the corresponding sensor data of the corresponding autonomous vehicle. Various implementations train a machine learning model based on such training instances. Once trained, the machine learning model can enable processing, using the machine learning model, of sensor data from a given autonomous vehicle to predict one or more properties of a given additional vehicle that is captured at least in part by the sensor data.

First claim

What is claimed is:

1. A method of training a machine learning model to be used in autonomous vehicle control, the method performed by one or more processors and comprising:
identifying instances of autonomous vehicle sensor data, wherein each of the instances of autonomous vehicle sensor data:
is generated based on corresponding output from one or more autonomous vehicle sensor components of an autonomous vehicle, and
includes a corresponding autonomous vehicle timestamp generated using an autonomous vehicle clock local to the autonomous vehicle;

identifying instances of additional vehicle data, wherein each of the instances of additional vehicle data:
is generated based on one or more additional vehicle sensor components of an additional vehicle, and
includes a corresponding additional vehicle timestamp generated using an additional vehicle clock local to the additional vehicle;

generating a plurality of training instances, generating each of the plurality of training instances comprising:
generating a corresponding training instance input, of the plurality of training instances, based on a corresponding one of the instances of autonomous vehicle sensor data, wherein the corresponding training instance input is processed using the machine learning model to predict at least a motion state of the additional vehicle; and

generating a corresponding supervised label, of the plurality of training instances, generating the corresponding supervised label comprising:
generating the corresponding supervised label that describes the motion state of the additional vehicle using a corresponding one of the instances of additional vehicle data based on determining that the corresponding additional vehicle timestamp, of the corresponding one of the instances of additional vehicle data, corresponds to the corresponding autonomous vehicle timestamp of the corresponding one of the instances of autonomous vehicle sensor data of the corresponding training instance input; and

training the machine learning model using the plurality of training instances.

2. The method of claim 1 , wherein the additional vehicle clock local to the additional vehicle is part of a computing device, of the additional vehicle, that logs the instances of additional vehicle data.

3. The method of claim 1 , further comprising: determining a delta between the autonomous vehicle clock and the additional vehicle clock; wherein determining that the corresponding additional vehicle timestamp corresponds to the corresponding autonomous vehicle timestamp comprises using the delta in determining that the corresponding additional vehicle timestamp corresponds to the corresponding autonomous vehicle timestamp.

4. The method of claim 1 , further comprising: prior to the instances of autonomous vehicle data and the instances of additional vehicle data being generated: synchronizing the autonomous vehicle clock and the additional vehicle clock.

5. The method of claim 1 , wherein the one or more autonomous vehicle sensor components of the autonomous vehicle comprise a vision component.

6. The method of claim 5 , wherein the vision component is a Light Detection and Ranging (LIDAR) component, a monographic camera, a stereographic camera, or a thermal camera.

7. The method of claim 6 , wherein the LIDAR component is a phase coherent LIDAR component.

8. The method of claim 1 , wherein training the machine learning model comprises: processing, using the machine learning model, the corresponding training instance inputs of the plurality of training instances to generate predicted outputs; generating losses based on comparing the predicted outputs to the corresponding supervised labels of the plurality of training instances; and updating the machine learning model based on the generated losses.

9. The method of claim 1 , further comprising providing the trained machine learning model for use in control of a given autonomous vehicle.

10. The method of claim 9 , further comprising: processing, by at least one processor of the given autonomous vehicle, given sensor data using the trained machine learning model, the given sensor data captured by given sensor components of the given autonomous vehicle; generating, based on the processing, a predicted output; and controlling the given autonomous vehicle based on the predicted output.

11. The method of claim 10 , wherein controlling the given autonomous vehicle comprises controlling speed or direction of the given autonomous vehicle or a combination of speed and direction.

12. The method of claim 10 , wherein controlling the given autonomous vehicle comprises performing a controlled stop of the given autonomous vehicle.

13. The method of claim 1 , wherein the corresponding supervised labels each comprise a corresponding bounding area.

14. The method of claim 1 , wherein the corresponding supervised labels each comprise one or more corresponding yaw parameters and the one or more corresponding sensor components of the additional vehicle comprise a yaw rate sensor.

15. The method of claim 14 , wherein the additional vehicle data is determined based on monitoring of a controller area network of the corresponding additional vehicle.

16. The method of claim 1 , wherein the corresponding supervised labels each comprise one or both of a corresponding velocity and a corresponding acceleration of the additional vehicle.

17. The method of claim 1 , wherein the corresponding supervised labels each comprise one or more corresponding yaw parameters and one or both of a corresponding velocity and a corresponding acceleration of the additional vehicle.

18. One or more non-transitory computer readable storage media comprising computer instructions executable by one or more processors that when executed cause the at least one processor to:
identify instances of autonomous vehicle sensor data, wherein each of the instances of autonomous vehicle sensor data:
is generated based on corresponding output from one or more autonomous vehicle sensor components of an autonomous vehicle, and
includes a corresponding autonomous vehicle timestamp generated using an autonomous vehicle clock local to the autonomous vehicle;

identify instances of additional vehicle data, wherein each of the instances of additional vehicle data:
is generated based on one or more additional vehicle sensor components of an additional vehicle, and
includes a corresponding additional vehicle timestamp generated using an additional vehicle clock local to the additional vehicle;

generate a plurality of training instances, generating each of the plurality of training instances comprising:
generate a corresponding training instance input, of the plurality of training instances, based on a corresponding one of the instances of autonomous vehicle sensor data, wherein the corresponding training instance input is processed using a machine learning model to predict at least a motion state of the additional vehicle; and

generate a corresponding supervised label, of the plurality of training instances, generating the corresponding supervised label comprising:
generate the corresponding supervised label that describes the motion state of the additional vehicle using a corresponding one of the instances of additional vehicle data based on determining that the corresponding additional vehicle timestamp, of the corresponding one of the instances of additional vehicle data, corresponds to the corresponding autonomous vehicle timestamp of the corresponding one of the instances of autonomous vehicle sensor data of the corresponding

Assignees

• Aurora Innovation Inc
• Aurora Operations Inc

Inventors

• Smith Warren
• Eade Ethan
• Anderson Sterling J
• Bagnell James Andrew
• Nabbe Bartholomeus C
• Urmson Christopher Paul

Classifications

• G06F18/2178

  based on feedback of a supervisor · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title