Labeling autonomous vehicle data

Patent metadata

Field	Value
Publication number	US-11630458-B2
Application number	US-202117553209-A
Country	US
Kind code	B2
Filing date	Dec 16, 2021
Priority date	Nov 2, 2018
Publication date	Apr 18, 2023
Grant date	Apr 18, 2023

Abstract

Sensor data collected from an autonomous vehicle data can be labeled using sensor data collected from an additional vehicle. The additional vehicle can include a non-autonomous vehicle mounted with a removable hardware pod. In many implementations, removable hardware pods can be vehicle agnostic. In many implementations, generated labels can be utilized to train a machine learning model which can generate one or more control signals for the autonomous vehicle.

First claim

The invention claimed is:

1. A method of generating labeled autonomous vehicle data, the method comprising:
receiving autonomous vehicle data collected using an autonomous vehicle sensor suite of an autonomous vehicle, wherein at least one instance of autonomous vehicle data includes an autonomous vehicle time stamp and at least one of the sensors of the autonomous vehicle sensor suite detects an additional vehicle in an environment;
receiving additional vehicle data collected using an additional vehicle sensor suite of the additional vehicle, wherein at least one instance of additional vehicle data has an additional vehicle time stamp;
temporally correlating one or more instances of autonomous vehicle data with one or more instances of the additional vehicle data using the autonomous vehicle time stamps and the additional vehicle time stamps, wherein temporally correlating the one or more instances of autonomous vehicle data with the one or more instances of additional vehicle data includes synchronizing vehicle time stamps between sets of data collected by the autonomous vehicle and the additional vehicle in the same environment;
generating a plurality of labels, wherein at least one label identifies a current state of at least one attribute of the additional vehicle determined using the one or more instances of the additional vehicle data temporally correlated with the one or more instances of autonomous vehicle data.

2. The method of claim 1 , wherein synchronizing the vehicle time stamps between the sets of data collected by the autonomous vehicle and the additional vehicle in the same environment comprises:
in response to the vehicle time stamps between the sets of data collected by the autonomous vehicle and the additional vehicle in the same environment at approximately same time being different, shifting the autonomous vehicle time stamps or shifting the additional vehicle time stamps.

3. The method of claim 2 , wherein:
the shifting includes matching the autonomous vehicle time stamps and the additional vehicle time stamps.

4. The method of claim 1 , further comprising:
in response to detecting that a first instance of the additional vehicle data includes a location of an object not captured or partially captured by the first instance of the autonomous vehicle data that is temporally correlated to the first instance of the additional vehicle data, determining the location of the object in the first instance of the autonomous vehicle data using (1) the first instance of the additional vehicle data, (2) a location of the autonomous vehicle in the first instance of the autonomous vehicle data, and (3) a location of the additional vehicle in the first instance of the autonomous vehicle data.

5. The method of claim 4 , further comprising:
in response to detecting that the first instance of the additional vehicle data includes information of the object other than the location of the object, mapping the information of the object to the location of the object in the first instance of the autonomous vehicle data.

6. The method of claim 1 , wherein:
a secondary label of the plurality of labels that identifies an attribute of the additional vehicle determined using a second instance of additional vehicle data is mapped to a position of the additional vehicle in a second instance of autonomous vehicle data, and
the second instance of autonomous vehicle data is temporally correlated with to the second instance of additional vehicle data.

7. The method of claim 6 , wherein the second instance of autonomous vehicle data and the second instance of additional vehicle data have a common time stamp.

8. The method of claim 1 , further comprising:
determining a proximity between the autonomous vehicle and the additional vehicle in the same environment, at least one autonomous vehicle time stamp and/or at least one additional vehicle time stamp.

9. The method of claim 8 , wherein prior to generating the plurality of labels, the method further comprises:
removing one or more instances of additional vehicle data that respectively includes an additional vehicle time stamp at which the proximity between the autonomous vehicle and the additional vehicle is beyond a proximity range.

10. The method of claim 8 , wherein prior to generating the plurality of labels, the method further comprises:
removing one or more instances of autonomous vehicle data that respectively includes the autonomous vehicle time stamp at which the proximity between the autonomous vehicle and the additional vehicle is beyond the proximity range.

11. The method of claim 8 , wherein the proximity is determined using one or more proximity sensors.

12. The method of claim 1 , wherein:
the additional vehicle is a non-autonomous vehicle, and
the additional vehicle sensor suite is a removable hardware pod connected to a CAN bus of the additional vehicle.

13. The method of claim 1 , wherein:
the autonomous vehicle time stamps include at least one autonomous vehicle time stamp that is different from any of the additional vehicle time stamps.

14. The method of claim 1 , wherein:
the autonomous vehicle time stamps are respectively added to corresponding instances of autonomous vehicle data prior to being uploaded for further processing.

15. The method of claim 1 , wherein:
the additional vehicle time stamps are respectively added to corresponding instances of additional vehicle data prior to being uploaded for further processing.

16. The method of claim 1 , wherein the additional vehicle is detected in one or more instances of autonomous vehicle data.

17. The method of claim 1 , wherein:
the autonomous vehicle data is time stamped using a printed circuit board (PCB) and/or a computing device coupled to the autonomous vehicle sensor suite.

18. The method of claim 1 , wherein:
the additional vehicle data is time stamped using a PCB or computing device coupled to the additional vehicle sensor suite, and/or using a sensor within the additional vehicle sensor suite.

19. A system comprising one or more processor and a memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of instructions by the one or more processors, cause the one or more processors to perform a method comprising:
receiving autonomous vehicle data collected using an autonomous vehicle sensor suite of an autonomous vehicle, wherein at least one instance of autonomous vehicle data includes an autonomous vehicle time stamp and at least one of the sensors of the autonomous vehicle sensor suite detects an additional vehicle in an environment;
receiving additional vehicle data collected using an additional vehicle sensor suite of the additional vehicle, wherein at least one instance of additional vehicle data has an additional vehicle time stamp;
temporally correlating one or more instances of autonomous vehicle data with one or more instances of the additional vehicle data using the autonomous vehicle time stamps and the additional vehicle time stamps, wherein temporally correlating the one or more instances of autonomous vehicle data with the one or more instances of additional vehicle data includes synchronizing vehicle time stamps between sets of data collected by the autonomous vehicle and the additional vehicle in the same environment;
generating a plurality of labels, wherein at least one label identifies a current state of at least one attribute of the additional vehicle determined using the one or more instances of the addit

Assignees

• Aurora Operations Inc

Inventors

• Valois Jean-Sebastien
• Eade Ethan

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G01S13/931

  of land vehicles · CPC title