Method and system for in-vehicle self-supervised training of perception functions for an automated driving system

Patent metadata

Field	Value
Publication number	US-12586382-B2
Application number	US-202318452045-A
Country	US
Kind code	B2
Filing date	Aug 18, 2023
Priority date	Aug 18, 2022
Publication date	Mar 24, 2026
Grant date	Mar 24, 2026

Abstract

A method for updating a perception function of a vehicle having an Automated Driving System (ADS) is disclosed. The ADS has a self-supervised machine-learning (ML) algorithm for reconstructing an ingested image and a ML algorithm for an in-vehicle perception module for detecting one or more objects or free-space areas depicted in an ingested image. At first, an image of a scene in a surrounding environment of the vehicle is obtained. The obtained image is processed to obtain an output image with one or more detected objects or free-space areas. Then, an evaluation dataset is formed accordingly. The evaluation dataset and the obtained image is processed to obtain a reconstruction error value for each evaluation image and an evaluation image with highest reconstruction error value is selected among plurality of evaluation images. Using the obtained image and the selected evaluation image, the ML algorithm for the in-vehicle perception module is updated.

First claim

The invention claimed is:

1 . A computer-implemented method for updating a perception function of a vehicle having an Automated Driving System (ADS), wherein the ADS comprises a self-supervised machine-learning algorithm configured to reconstruct an ingested image and a machine-learning algorithm for an in-vehicle perception module configured to detect one or more objects or free-space areas depicted in an ingested image, the method comprising:
obtaining, by a processing unit, an image of a scene in a surrounding environment of the vehicle; processing, by the processing unit, the obtained image, by the machine-learning algorithm for the in-vehicle perception module, in order to obtain an output image with one or more detected objects or free-space areas, each detected object or free-space area is indicated in the output image; forming, by the processing unit, an evaluation dataset based on the obtained output image, wherein the evaluation dataset comprises a plurality of evaluation images corresponding to the scene comprised in the obtained image with masked permutations of the indications of the one or more detected objects or free-space areas in the output image; processing, by the processing unit, the evaluation dataset and the obtained image, by means of the self-supervised machine-learning algorithm, in order to obtain a reconstruction error value for each evaluation image; selecting, by the processing unit, an evaluation image out of the plurality of evaluation images, wherein the selected evaluation image has the highest reconstruction error value under a regularization restriction defined based on at least a size of the masked and permuted indications; and updating, by the processing unit, the machine-learning algorithm for the in-vehicle perception module based on the obtained image and the selected evaluation image, wherein the permuted indication of the selected evaluation image defines a ground truth.

2 . The computer-implemented method according to claim 1 , wherein each detected object or free-space area is indicated by at least one of:
a bounding box, and a masking region.

3 . The computer-implemented method according to claim 1 , wherein the indications are permuted differently in each evaluation image.

4 . The computer-implemented method according to claim 1 , wherein the forming the evaluation dataset comprises forming the plurality of evaluation images, and for each evaluation image:
permuting and masking, by the processing unit, each indication of the output image such that each evaluation image comprises the scene comprised in the obtained image with masked permutations of the indications of the one or more detected objects or free-space areas in the output image and such that each evaluation image differs from the other evaluation images of the plurality of evaluation images.

5 . The computer-implemented method according to claim 4 , wherein the permutation of each indication comprises at least one of changing a size of the indication, changing a shape of the indication, and changing a location of the indication.

6 . The computer-implemented method according to claim 5 , wherein the indication is in the form of a bounding box, and wherein the changing the shape of the indication comprises changing a width-to-height ratio of the bounding box.

7 . The computer-implemented method according to claim 1 , wherein the self-supervised machine-learning algorithm comprises a Masked Autoencoder (MAE).

8 . The computer-implemented method according to claim 1 , further comprising:
detecting, by the processing unit, anomalous image data using a machine-learning classification system trained to distinguish new experiences from experiences known to the self-supervised machine-learning algorithm in the obtained image and to output an anomaly value; wherein the updating the machine-learning algorithm for the in-vehicle perception module is only performed if the anomaly value is below a threshold.

9 . The computer-implemented method according to claim 1 , further comprising:
updating, by the processing unit, one or more model parameters of the self-supervised machine-learning algorithm in accordance with a self-supervised machine learning process based on the obtained image.

10 . The computer-implemented method according to claim 9 , further comprising:
transmitting, by the processing unit, the updated one or more model parameters of the self-supervised machine-learning algorithm and the updated one or more model parameters of the machine-learning algorithm for the in-vehicle perception module to a remote entity; receiving, by the processing unit, a set of globally updated one or more model parameters of the self-supervised machine-learning algorithm from the remote entity, wherein the set of globally updated one or more model parameters of the self-supervised machine-learning algorithm are based on information obtained from a plurality of vehicles comprising a corresponding self-supervised machine-learning algorithm; receiving, by the processing unit, a set of globally updated one or more model parameters of the machine-learning algorithm for the in-vehicle perception module from the remote entity, wherein the set of globally updated one or more model parameters of the machine-learning algorithm for the in-vehicle perception module are based on information obtained from a plurality of vehicles comprising a corresponding machine-learning algorithm for the in-vehicle perception module; updating, by the processing unit, the self-supervised machine-learning algorithm based on the received set of globally updated one or more model parameters of the self-supervised machine-learning algorithm; and updating, by the processing unit, the machine-learning algorithm for the in-vehicle perception module based on the received set of globally updated one or more model parameters of the machine-learning algorithm for the in-vehicle perception module.

11 . A non-transitory computer-readable storage medium comprising instructions which, when executed by a computing device, causes the computing device to carry out method according to claim 1 .

12 . A system for updating a perception function of a vehicle having an Automated Driving System (ADS), wherein the ADS comprises a self-supervised machine-learning algorithm configured to reconstruct an ingested image and a machine-learning algorithm for an in-vehicle perception module configured to detect one or more objects or free-space areas depicted in an ingested image, the system comprising control circuitry configured to:
obtain an image of a scene in a surrounding environment of the vehicle; process the obtained image by the machine-learning algorithm for the in-vehicle perception module, in order to obtain an output image with one or more detected objects or free-space areas, each detected object or free-space area is indicated in the output image; form an evaluation dataset based on the obtained output image, wherein the evaluation dataset comprises a plurality of evaluation images corresponding to the scene comprised in the obtained image with masked permutations of the indications of the one or more detected objects or free-space areas in the output image; process the evaluation dataset and the obtained image, by means of the self-supervised machine-learning algorithm, in order to obtain a reconstruction error value for each evaluation image; select an evaluation image out of the plurality of evaluation images, wherein the selected evaluation image has the highest reconstruction error value under a regularization restriction defined based on at least a size of the masked and permuted indications; an

Assignees

• Zenseact Ab

Inventors

• Gyllenhammar Magnus
• Tonderski Adam

Classifications

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/7792

  the supervisor being an automated module, e.g. "intelligent oracle" · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title