Generating labeled training instances for autonomous vehicles using temporally correlated timestamps

Patent metadata

Field	Value
Publication number	US-12572809-B2
Application number	US-202217713782-A
Country	US
Kind code	B2
Filing date	Apr 5, 2022
Priority date	Nov 2, 2018
Publication date	Mar 10, 2026
Grant date	Mar 10, 2026

Abstract

In techniques disclosed herein, machine learning models can be utilized in the control of autonomous vehicle(s), where the machine learning models are trained using automatically generated training instances. In some such implementations, a label corresponding to an object in a labeled instance of training data can be mapped to the corresponding instance of unlabeled training data. For example, an instance of sensor data can be captured using one or more sensors of a first sensor suite of a first vehicle can be labeled. The label(s) can be mapped to an instance of data captured using one or more sensors of a second sensor suite of a second vehicle.

First claim

What is claimed:

1 . A method implemented at a remote computing system for generating labeled sensor data for training a machine learning model of an autonomous vehicle, the method comprising:
receiving first sensor data collected using a first vehicle sensor suite of a first vehicle, wherein the first sensor data comprises first vehicle time stamps and wherein at least a portion of the first sensor data comprises a representation of an additional object in an environment,
wherein one or more first vehicle time stamps are respectively added to the one or more instances of first sensor data prior to the one or more instances of first sensor data being uploaded for further processing;
receiving second sensor data collected using a second vehicle sensor suite of a second vehicle, wherein the second sensor data comprises second vehicle time stamps; temporally correlating one or more instances of the first sensor data with one or more instances of second sensor data using the first vehicle time stamps and the second vehicle time stamps; generating a label for the first sensor data that identifies a current state of at least one attribute of the additional object, that is at least partially occluded in the first sensor data, determined using the one or more instances of second sensor data temporally correlated with the one or more instances of first sensor data; processing, using the machine learning model, the first sensor data to generate a predicted label that is predicted to identify the current state of the at least one attribute of the additional object; and updating, based on a difference between the label and the predicted label, one or more weights of the machine learning model.

2 . The method of claim 1 , wherein:
the first vehicle is an autonomous vehicle, the second vehicle is a non-autonomous vehicle, and the second vehicle sensor suite is a removable hardware pod.

3 . The method of claim 1 , wherein:
one or more second vehicle time stamps are respectively added to the one or more instances of second sensor data prior to the one or more instances of second sensor data being uploaded for further processing.

4 . The method of claim 1 , wherein the second vehicle is captured in the one or more instances of first sensor data.

5 . The method of claim 1 , wherein:
the first sensor data is time stamped using a printed circuit board (PCB) and/or a computing device coupled to the first vehicle sensor suite.

6 . The method of claim 1 , wherein:
the second sensor data is time stamped using a PCB or computing device coupled to the second vehicle sensor suite, and/or using a sensor within the second vehicle sensor suite.

7 . A remote system comprising one or more processors and a memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to perform a method comprising:
receiving first sensor data collected using a first vehicle sensor suite of a first vehicle, wherein the first sensor data comprises first vehicle time stamps and wherein at least a portion of the first sensor data comprises a representation of an additional object in an environment,
wherein one or more first vehicle time stamps are respectively added to the one or more instances of first sensor data prior to the one or more instances of first sensor data being uploaded for further processing;
receiving second sensor data collected using a second vehicle sensor suite of a second vehicle, wherein the second sensor data comprises second vehicle time stamps; temporally correlating one or more instances of the first sensor data with one or more instances of second sensor data using the first vehicle time stamps and the second vehicle time stamps; generating a label for the first sensor data that identifies a current state of at least one attribute of the additional object, that is at least partially occluded in the first sensor data, determined using the one or more instances of second sensor data temporally correlated with the one or more instances of first sensor data; processing, using a machine learning model, the first sensor data to generate a predicted label that is predicted to identify the current state of the at least one attribute of the additional object; and updating, based on a difference between the label and the predicted label, one or more weights of the machine learning model.

8 . The remote system of claim 7 , wherein:
the first vehicle is an autonomous vehicle, the second vehicle is a non-autonomous vehicle, and the second vehicle sensor suite is a removable hardware pod.

9 . The remote system of claim 7 , wherein:
one or more second vehicle time stamps are respectively added to the one or more instances of second sensor data prior to the one or more instances of second sensor data being uploaded for further processing.

10 . The remote system of claim 7 , wherein the second vehicle is captured in the one or more instances of first sensor data.

11 . The remote system of claim 7 , wherein:
the first sensor data is time stamped using a printed circuit board (PCB) and/or a computing device coupled to the first vehicle sensor suite.

12 . The remote system of claim 7 , wherein:
the second sensor data is time stamped using a PCB or computing device coupled to the second vehicle sensor suite, and/or using a sensor within the second vehicle sensor suite.

13 . A non-transitory computer readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method at a remote computing system, the method comprising:
receiving first sensor data collected using a first vehicle sensor suite of a first vehicle, wherein the first sensor data comprises first vehicle time stamps and wherein at least a portion of the first sensor data comprises a representation of an additional object in an environment,
wherein one or more first vehicle time stamps are respectively added to the one or more instances of first sensor data prior to the one or more instances of first sensor data being uploaded for further processing;
receiving second sensor data collected using a second vehicle sensor suite of a second vehicle, wherein the second sensor data comprises second vehicle time stamps; temporally correlating one or more instances of the first sensor data with one or more instances of second sensor data using the first vehicle time stamps and the second vehicle time stamps; generating a label for the first sensor data that identifies a current state of at least one attribute of the additional object, that is at least partially occluded in the first sensor data, determined using the one or more instances of second sensor data temporally correlated with the one or more instances of first sensor data; processing, using a machine learning model, the first sensor data to generate a predicted label that is predicted to identify the current state of the at least one attribute of the additional object; and updating, based on a difference between the label and the predicted label, one or more weights of the machine learning model.

14 . The non-transitory computer readable storage medium of claim 13 , wherein:
the first vehicle is an autonomous vehicle, the second vehicle is a non-autonomous vehicle, and the second vehicle sensor suite is a removable hardware pod.

15 . The non-transitory computer readable storage medium of claim 13 , wherein:
one or more second vehicle time stamps

Assignees

• Aurora Operations Inc

Inventors

• Valois Jean-Sebastien
• Pilarski Thomas
• Munoz Daniel

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• B60W60/0025

  specially adapted for specific operations · CPC title