Method and system for integrated global and distributed learning in autonomous driving vehicles

Patent metadata

Field	Value
Publication number	US-11550334-B2
Application number	US-201715856332-A
Country	US
Kind code	B2
Filing date	Dec 28, 2017
Priority date	Jun 6, 2017
Publication date	Jan 10, 2023
Grant date	Jan 10, 2023

Abstract

The present teaching relates to system, method, medium for in-situ perception in an autonomous driving vehicle. A plurality of types of sensor data are received, which are acquired by a plurality of types of sensors deployed on the vehicle to provide information about surrounding of the vehicle. Based on at least one model, one or more surrounding items are tracked from a first of the plurality of types of sensor data acquired by a first type sensors. At least some of the tracked items are automatically labeled via cross validation and are used to locally adapt, on-the-fly, the at least one model. Model update information is received which from a model update center, which is derived based on the labeled at least some items. The at least one model is updated using the model update information.

First claim

We claim:

1. A method, comprising:
receiving sensor data acquired continuously by a plurality of sensors deployed on an autonomous driving vehicle, each sensor from the plurality of sensors having a sensor type from a plurality of sensor types, the sensor data providing information about a region surrounding the autonomous driving vehicle;
tracking, based on at least one object detection model residing in the autonomous driving vehicle, (1) one or more items in the region surrounding the autonomous driving vehicle, and (2) one or more features associated with at least one item from the one or more items, based on a first subset of the sensor data, the first subset of the sensor data being acquired by a first subset of at least one sensor from the plurality of sensors, the first subset of at least one sensor being of a first sensor type from the plurality of sensor types, the first subset of the sensor data being in a first modality;
labeling, automatically on-the-fly, the one or more items and the one or more features via cross modality validation of the one or more items and the one or more features based on a second subset of the sensor data in a second modality, the second subset of the sensor data being acquired by a second subset of at least one sensor from the plurality of sensors, the second subset of at least one sensor being of a second sensor type from the plurality of sensor types;
selecting, from the one or more items and after the labeling, a plurality of events of interest based on a mode of local model adaptation, the mode of local model adaptation being at least one of discrepancy based adaptation or reinforcement based adaptation; and
locally adapting, on-the-fly, the at least one object detection model based on at least one event of interest from the plurality of events of interest, the locally adapting including:
selecting, from the labeled one or more items, a first group of labeled items, each labeled item from the first group of labeled items indicating a discrepancy between a label of the labeled one or more items and a state of the one or more items in the region surrounding the autonomous driving vehicle;
selecting, from the labeled one or more items, a second group of labeled items, each labeled item from the second group of labeled items indicating a consistency between a label of the labeled one or more items and a state of the labeled one or more items in the region surrounding the autonomous driving vehicle;

determining an adaptation mode based on a model adaptation configuration; and
modifying the at least one object detection model based on the adaptation mode and at least one of:
the first group of labeled items,
the second group of labeled items, or
the first group of labeled items and the second group of labeled items; and
further comprising labeling the one or more items via cross temporal validation which comprises, for each of the one or more items:
determining an estimated label for that item;
retrieving previously labeled items associated with that item, each of the previously labeled items having an associated previous label from a plurality of previous labels;
assessing consistency between the plurality of previous labels and the estimated label;
if the estimated label is consistent with the plurality of previous labels, assigning the estimated label to that item; and
if the estimated label is inconsistent with the plurality of previous labels:
assigning the estimated label to that item and the previously labeled items if the estimated label is determined to be able to resolve the inconsistency,
assigning the previous labels to that item if the previous labels are able to resolve the inconsistency, and
assigning the estimated label to that item when the inconsistency is not resolved.

2. The method of claim 1 , wherein each of the one or more items is labeled as one of:
an object appearing in the region surrounding the autonomous driving vehicle;
a non-object; or
a non-conclusive item,
the label for each item from the one or more items having an associated confidence metric and an associated time.

3. The method of claim 1 , wherein the labeling via cross modality validation includes:
obtaining the second subset of the sensor data; and
for each of the one or more items and at each point of time from a plurality of points of time:
generating validation base data based on the second subset of the sensor data obtained at that point of time,
identifying a portion of the validation base data associated with that item,
cross validating, on-the-fly, that item based on the portion of the validation base data to generate an associated cross modality validation result,
and
labeling that item with a label based on the cross modality validation result and a time stamp consistent with that point of time.

4. The method of claim 1 , wherein, for each of the one or more items, the estimated label of that item is determined based on a result of cross modality validation with respect to that item.

5. The method of claim 1 , further comprising
sending, via a communication platform, a representation of at least one event of interest from the plurality of events of interest to a model update center to cause the model update center to derive model update information based on the at least one event of interest from the plurality of events of interest;
receiving, from the model update center, the model update information; and
updating the at least one object detection model based on the model update information.

6. A non-transitory, processor-readable medium storing instructions to cause a processor to:
receive sensor data acquired continuously by a plurality of sensors deployed on an autonomous driving vehicle, each sensor from the plurality of sensors having a sensor type from a plurality of sensor types, the sensor data providing information about a region surrounding the autonomous driving vehicle;
track based on at least one object detection model residing in the autonomous driving vehicle, (1) one or more items in the region surrounding the autonomous driving vehicle, and (2) one or more features associated with at least one item from the one or more items, based on a first subset of the sensor data, the first subset of the sensor data being acquired by a first subset of at least one sensor from the plurality of sensors, the first subset of at least one sensor being of a first sensor type from the plurality of sensor types, the first subset of the sensor data being in a first modality;
label, automatically on-the-fly, the one or more items and the one or more features via cross modality validation of the one or more items and the one or more features based on a second subset of the sensor data in a second modality, the second subset of the sensor data being acquired by a second subset of at least one sensor from the plurality of sensors, the second subset of at least one sensor being of a second sensor type from the plurality of sensor types;
select, from the one or more items and after the labeling, a plurality of events of interest based on a mode of local model adaptation, the mode of local model adaptation being at least one of discrepancy based adaptation or reinforcement based adaptation;
locally adapt, on-the-fly, the at least one object detection model based on at least one event of interest from the plurality of events of interest, by;
selecting, from the labeled one or more items, a first group of labeled items, each labeled item from the first group of labeled items indicating a discrepancy between a label of the labeled one or more items and a state of the one or more items in the region surrounding the autonomous driving vehicle;
selecting, from the labeled one or more items, a second group of labeled items, ea

Assignees

• Plusai Inc

Inventors

• Zheng Hao
• Liu David Wanqian
• Daly Timothy Patrick

Classifications

• G06V30/19147

  Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/98

  Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns · CPC title

• G06F18/24143

  Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN] · CPC title