Method and device for providing personalized and calibrated adaptive deep learning model for the user of an autonomous vehicle

Patent metadata

Field	Value
Publication number	US-10824151-B2
Application number	US-202016738320-A
Country	US
Kind code	B2
Filing date	Jan 9, 2020
Priority date	Jan 31, 2019
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

A method for providing a dynamic adaptive deep learning model other than a fixed deep learning model, to thereby support at least one specific autonomous vehicle to perform a proper autonomous driving according to surrounding circumstances is provided. And the method includes steps of: (a) a managing device which interworks with autonomous vehicles instructing a fine-tuning system to acquire a specific deep learning model to be updated; (b) the managing device inputting video data and its corresponding labeled data to the fine-tuning system as training data, to thereby update the specific deep learning model; and (c) the managing device instructing an automatic updating system to transmit the updated specific deep learning model to the specific autonomous vehicle, to thereby support the specific autonomous vehicle to perform the autonomous driving by using the updated specific deep learning model other than a legacy deep learning model.

First claim

What is claimed is:

1. A method for providing a dynamic adaptive deep learning model other than a fixed deep learning model, to thereby support at least one specific autonomous vehicle to perform a proper autonomous driving according to surrounding circumstances, comprising steps of:
(a) a managing device which interworks with autonomous vehicles, if a video data transmitted from the specific autonomous vehicle among the autonomous vehicles is acquired through a video storage system, instructing a fine-tuning system to acquire a specific deep learning model to be updated by using the video data from a deep learning model library storing one or more deep learning models;
(b) the managing device inputting the video data and its corresponding labeled data to the fine-tuning system as training data, to thereby update the specific deep learning model; and
(c) the managing device instructing an automatic updating system to transmit the updated specific deep learning model to the specific autonomous vehicle, to thereby support the specific autonomous vehicle to perform the autonomous driving by using the updated specific deep learning model other than a legacy deep learning model,
wherein the managing device acquires the labeled data by inputting the video data to at least one of an auto-labeling system and a manual-labeling system, and
wherein the auto-labeling system applies an auto-labeling operation, using a certain deep learning model for labeling acquired from the deep learning model library, to the video data, to thereby generate at least part of the labeled data, and the manual-labeling system distributes the video data to each of labelers by using a distribution algorithm and acquires outputs of the labelers corresponding to the video data, to thereby generate at least part of the labeled data.

2. The method of claim 1 , wherein, at the step of (b), the fine-tuning system instructs (i) at least one convolutional layer in the specific deep learning model to apply at least one convolutional operation to the video data, to thereby generate a feature map, (ii) at least one output layer therein to apply at least one output operation to the feature map, to thereby generate an estimated data, and (iii) at least one loss layer to generate a loss by referring to the estimated data and the labeled data and perform backpropagation by using the loss, to thereby fine-tune parameters of the specific deep learning model.

3. The method of claim 2 , wherein the managing device updates the specific deep learning model by fine-tuning the parameters of the specific deep learning model, to thereby support the specific deep learning model to perform the autonomous driving by referring to both a personalized optimal policy for the specific autonomous vehicle corresponding to the video data and a preset optimal policy.

4. The method of claim 1 , wherein the managing device instructs (i) a label-validating system to perform a cross-validation by comparing each of parts of the labeled data generated by each of the auto-labeling system and the manual-labeling system, to thereby generate feedback information, (ii) the auto-labeling system and the manual-labeling system to determine whether to adjust said parts of the labeled data by using the feedback information and adjust said parts of the labeled data when the feedback information indicates a necessity of an adjustment, (iii) and the label-validating system to deliver the labeled data which has been validated by the label-validating system to the fine-tuning system.

5. The method of claim 1 , wherein the manual-labeling system performs the distribution algorithm by referring to each of at least part of characteristics of each of the labelers including at least part of workload information and efficiency information thereof.

6. A method for providing a dynamic adaptive deep learning model other than a fixed deep learning model, to thereby support at least one specific autonomous vehicle to perform a proper autonomous driving according to surrounding circumstances, comprising steps of:
(a) a managing device which interworks with autonomous vehicles, if a video data transmitted from the specific autonomous vehicle among the autonomous vehicles is acquired through a video storage system, instructing a fine-tuning system to acquire a specific deep learning model to be updated by using the video data from a deep learning model library storing one or more deep learning models;
(b) the managing device inputting the video data and its corresponding labeled data to the fine-tuning system as training data, to thereby update the specific deep learning model; and
(c) the managing device instructing an automatic updating system to transmit the updated specific deep learning model to the specific autonomous vehicle, to thereby support the specific autonomous vehicle to perform the autonomous driving by using the updated specific deep learning model other than a legacy deep learning model,
wherein the video storage system applies at least one prevention operation to the video data, to thereby remove private information including at least part of portrait information of people and ID information of vehicles in the video data, and then stores the video data after encrypting it.

7. A method for providing a dynamic adaptive deep learning model other than a fixed deep learning model, to thereby support at least one specific autonomous vehicle to perform a proper autonomous driving according to surrounding circumstances, comprising steps of:
(a) a managing device which interworks with autonomous vehicles, if a video data transmitted from the specific autonomous vehicle among the autonomous vehicles is acquired through a video storage system, instructing a fine-tuning system to acquire a specific deep learning model to be updated by using the video data from a deep learning model library storing one or more deep learning models;
(b) the managing device inputting the video data and its corresponding labeled data to the fine-tuning system as training data, to thereby update the specific deep learning model; and
(c) the managing device instructing an automatic updating system to transmit the updated specific deep learning model to the specific autonomous vehicle, to thereby support the specific autonomous vehicle to perform the autonomous driving by using the updated specific deep learning model other than a legacy deep learning model,
wherein, at the step of (a), the managing device instructs the deep learning model library to find at least one among the deep learning models whose relationship score in relation to the video data is larger than a threshold, and to deliver it to the fine-tuning system as the specific deep learning model, and wherein relationship scores are calculated by using at least part of video subject vehicle information, video subject time information, video subject location information and video subject driver information of the deep learning models.

8. A method for providing a dynamic adaptive deep learning model other than a fixed deep learning model, to thereby support at least one specific autonomous vehicle to perform a proper autonomous driving according to surrounding circumstances, comprising steps of:
(a) a managing device which interworks with autonomous vehicles, if a video data transmitted from the specific autonomous vehicle among the autonomous vehicles is acquired through a video storage system, instructing a fine-tuning system to acquire a specific deep learning model to be updated by using the video data from a deep learning model library storing one or more deep learning models;
(b) the managing device inputting the video data and its corresponding labeled data to the fine-tuning system as training data, to thereby upda

Assignees

• Stradvision Inc

Inventors

• Kim Kye-Hyeon
• Kim Yongjoong
• Kim Hak-Kyoung
• Nam Woonhyun
• Boo Sukhoon
• Sung Myungchul
• Shin Dongsoo
• Yeo Donghun
• Ryu Wooju
• Lee Myeong-Chun
• Lee Hyungsoo
• Jang Taewoong
• Jeong Kyungjoong
• Je Hongmo
• Cho Hojin

Classifications

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• B60W60/0011

  involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles · CPC title