Generating targeted training instances for autonomous vehicles

Patent metadata

Field	Value
Publication number	US-11256263-B2
Application number	US-201916271628-A
Country	US
Kind code	B2
Filing date	Feb 8, 2019
Priority date	Nov 2, 2018
Publication date	Feb 22, 2022
Grant date	Feb 22, 2022

Abstract

Sensor data collected via an autonomous vehicle can be labeled using sensor data collected via an additional vehicle, such as a non-autonomous vehicle mounted with a vehicle agnostic removable hardware pod. A training instance can include an instance of data collected by an autonomous vehicle sensor suite and one or more corresponding labels.

First claim

What is claimed:

1. A method of training a machine learning model with targeted training instances to be used in autonomous control of at least one autonomous vehicle, the method comprising:
generating a plurality of targeted training instances, wherein generating each of the targeted training instances includes:
generating autonomous vehicle training input of the targeted training instance based on an instance of autonomous vehicle data wherein at least one of the sensors of an autonomous vehicle sensor suite detects an additional vehicle in the environment;
generating a label of the autonomous vehicle training input indicating a current state of at least one attribute of the additional vehicle using a determined corresponding instance of additional vehicle data captured using an additional vehicle sensor suite of the additional vehicle,
wherein the instance of additional vehicle data is temporally correlated with the instance of autonomous vehicle data detecting the additional vehicle,
wherein at least one of the sensors in the additional vehicle sensor suite detects the at least one attribute of the additional vehicle, and
wherein the additional vehicle sensor suite is a removable hardware pod;

generating the trained machine learning model by:
applying the autonomous vehicle training input of the targeted training instance as training input to the machine learning model to generate predicted output of the machine learning model; and
updating one or more weights in the machine learning model by determining a difference between the predicted output and the label of the targeted training instance.

2. The method of claim 1 , wherein the machine learning model is a neural network model, and further comprising:
generating the trained neural network model using supervised learning by:
applying the autonomous vehicle data portion of the targeted training instance as training input to the neural network model to generate predicted output of the neural network model; and
updating, using backpropagation, the one or more weights in the neural network model by determining a difference between the predicted output and the label of the targeted training instance.

3. The method of claim 2 , further comprising: providing the trained neural network model for use in control of the autonomous vehicle.

4. The method of claim 1 , wherein the additional vehicle is a second autonomous vehicle.

5. The method of claim 1 , wherein the autonomous vehicle sensor suite comprises at least a Global Positioning System (GPS) unit, a radio direction and ranging (RADAR) unit, a light detection and ranging (LIDAR) unit, one or more cameras, and an inertial measurement (IMU) unit.

6. The method of claim 5 , wherein autonomous vehicle data comprises at least GPS data, RADAR data, LIDAR data, one or more images from the one or more cameras, and IMU data.

7. The method of claim 1 , wherein the additional vehicle is a non-autonomous vehicle.

8. The method of claim 7 , wherein the removable hardware pod is mounted onto the additional vehicle.

9. The method of claim 8 , wherein the removable hardware pod comprises at least a Global Positioning System (GPS) unit, a light detection and ranging (LIDAR) unit, and one or more cameras.

10. The method of claim 9 , wherein additional vehicle data comprises at least GPS data, LIDAR data, one or more images from one or more cameras, IMU data, and known additional vehicle data.

11. The method of claim 10 , wherein known additional vehicle data is selected from a group consisting of a vehicle make, a vehicle model, a vehicle color, a vehicle year, one or more vehicle dimension measurements, and a position of where the removable hardware pod is mounted on the additional vehicle, and combinations thereof.

12. The method of claim 10 , further comprising:
determining a location of the autonomous vehicle for each instance of the autonomous vehicle data utilizing GPS data from the one or more sensors of the autonomous vehicle; and
determining a location of the additional vehicle in each instance of autonomous vehicle data utilizing GPS data from the corresponding instance of additional vehicle data.

13. The method of claim 10 , further comprising:
determining a location of the autonomous vehicle in each instance of autonomous vehicle data and a location of the additional vehicle detected by at least one of the sensors of the autonomous vehicle in the autonomous vehicle data by localizing a location of the autonomous vehicle and a location of the additional vehicle using one or more common landmarks identified in the autonomous vehicle data and the corresponding instance of additional vehicle data.

14. The method of claim 10 , further comprising:
determining a bounding box indicating the location of the additional vehicle detected in the autonomous vehicle data.

15. The method of claim 14 , wherein determining the bounding box indicating the location of the additional vehicle detected in the autonomous vehicle data comprises:
determining the bounding box indicating the location of the additional vehicle detected in the autonomous vehicle data utilizing the one or more vehicle dimension measurements, the position where the removable hardware pod is mounted onto the additional vehicle, and a determined distance between the autonomous vehicle and the additional vehicle.

16. The method of claim 7 , wherein the additional vehicle is selected from a group consisting of a car, a van, a truck, a bus, a motorcycle, and a tractor trailer.

17. The method of claim 7 , wherein generating the trained machine learning model further comprises: not overfitting the machine learning model to the removable hardware pod.

18. The method of claim 17 , where not overfitting the machine learning model to the removable hardware pod comprises utilizing a plurality of shapes for an enclosure for the one or more sensors of the removable hardware pod.

19. The method of claim 17 , wherein not overfitting the machine learning model to the removable hardware pod comprises utilizing a plurality of mounting positions to mount the removable hardware pod onto the additional vehicle.

20. The method of claim 17 , wherein not overfitting the machine learning model to the removable hardware pod comprises masking out the removable hardware pod mounted onto the additional vehicle in each instance of autonomous vehicle data detecting the additional vehicle using one or more image processing techniques.

21. The method of claim 1 , further comprising:
generating additional output by processing an instance of second autonomous vehicle data from at least one sensor of a second autonomous vehicle sensor suite of a second autonomous vehicle using the machine learning model trained using one or more targeted training instances of the autonomous vehicle; and
generating one or more control commands for control of the autonomous vehicle based on the additional output.

22. A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to the execution of the instructions by one or more processors, cause the one or more processors to perform the following operations:
generating a plurality of targeted training instances, wherein generating each of the targeted training instances includes:
generating autonomous vehicle training input of the targeted training instance based on an instance of autonomous vehicle data wherein

Assignees

• Aurora Innovation Inc
• Aurora Operations Inc

Inventors

• Valois Jean-Sebastien
• Pilarski Thomas
• Munoz Daniel

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title