Image processing apparatus and image processing method

Patent metadata

Field	Value
Publication number	US-12430894-B2
Application number	US-202217942815-A
Country	US
Kind code	B2
Filing date	Sep 12, 2022
Priority date	Mar 11, 2022
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

An image processing apparatus has a first image acquisitor that acquires a source image, a second image acquisitor that acquires a first target image, a label acquisitor that acquires a label, a feature extractor including a first neural network that extracts a feature of the source image and a feature of the first target image, a class classifier including a second neural network that performs a class classification of the source image and the first target image, a domain classifier including a third neural network that performs a domain classification of the source image and the first target image, a processor that assigns a pseudo label to the first target image, a self-learner that performs a self-learning of the first neural network, the second neural network, and the third neural network, and a learner that learns the first, second and third neural networks, by performing a back propagation process.

First claim

The invention claimed is:

1. An image processing apparatus comprising:
a first image acquisitor configured to acquire a source image to which a label is assigned;
a second image acquisitor configured to acquire a first target image to which no label is assigned;
a label acquisitor configured to acquire a label;
a feature extractor including a first neural network configured to extract a feature of the source image and a feature of the first target image;
a class classifier including a second neural network configured to perform a class classification of the source image and the first target image based on a plurality of the features extracted by the feature extractor;
a domain classifier including a third neural network configured to perform a domain classification of the source image and the first target image based on the feature extracted by the feature extractor;
a processor configured to assign a pseudo label to the first target image using the class classifier including the second neural network in a middle of learning;
a self-learner configured to perform a self-learning of the first neural network, the second neural network, and the third neural network based on a feature obtained by inputting the first target image to which the pseudo label is assigned to the feature extractor; and
a learner configured to learn the first neural network, the second neural network, and the third neural network by performing a back propagation process based on a classification result by the class classifier, a classification result by the domain classifier, and a self-learning result by the self-learner.

2. The image processing apparatus according to claim 1 , further comprising:
a first loss calculator configured to calculate a first loss representing reliability of the class classification of the source image by the class classifier;
a second loss calculator configured to calculate a second loss representing reliability of the domain classification of the source image and the first target image by the domain classifier; and
a third loss calculator configured to calculate a third loss representing reliability of the class classification of the first target image to which the pseudo label is assigned, wherein
the self-learner is configured to input the first target image to which the pseudo label is assigned to the feature extractor and is configured to cause the third loss calculator to calculate the third loss, and
the learner is configured to learn the first neural network, the second neural network, and the third neural network by performing the back propagation process based on the first loss, the second loss, and the third loss.

3. The image processing apparatus according to claim 1 , further comprising:
a first loss calculator configured to calculate a first loss representing reliability of the class classification of the source image by the class classifier;
a second loss calculator configured to calculate a second loss representing reliability of the domain classification of the source image and the first target image by the domain classifier; and
a third loss calculator configured to calculate a third loss representing reliability of the class classification of the first target image by the class classifier, wherein
the self-learner is configured to input the first target image to which the pseudo label is assigned to the feature extractor and is configured to cause the third loss calculator to calculate the third loss, and
the learner comprises a first learner configured to perform the back propagation process based on the first loss and the second loss to learn the first neural network, the second neural network, and the third neural network, and a second learner configured to perform the back propagation process based on the third loss to learn the first neural network and the second neural network.

4. The image processing apparatus according to claim 2 , wherein
the self-learner is configured to stop the self-learning when the second loss is equal to or greater than a threshold value after starting the self-learning.

5. The image processing apparatus according to claim 2 , further comprising:
a weight loss generator configured to adjust weights of the first loss, the second loss, and the third loss when the back propagation process is performed to generate a weight loss, wherein
the learner is further configured to perform the back propagation process based on the weight loss to learn the first neural network, the second neural network, and the third neural network.

6. The image processing apparatus according to claim 5 , wherein
the weight loss generator is configured to generate the weight loss by lowering weights of the first loss and the second loss and raising a weight of the third loss as the self-learning by the self-learner progresses.

7. The image processing apparatus according to claim 2 , wherein
with an image group including a plurality of the source images and a plurality of the first target images being set as one epoch, the class classifier and the domain classifier are configured to perform class classification and domain classification for a plurality of epochs, and
the self-learner is configured to start labeling the first target image and the self-learning in a case where the first loss is equal to or less than a threshold value or in a case where the number of the processed epochs exceeds a predetermined ratio with respect to a total number of the epochs.

8. The image processing apparatus according to claim 1 , further comprising:
a third image acquisitor configured to acquire a second target image to which a label is assigned, the number of the second target images being smaller than the number of the first target images, wherein
the feature extractor is configured to extract a feature of the second target image,
the class classifier is configured to perform the class classification of the second target image based on the feature of the second target image extracted by the feature extractor,
the domain classifier is configured to perform the domain classification of the second target image based on the feature of the second target image extracted by the feature extractor, and
the learner is configured to learn the first neural network, the second neural network, and the third neural network based on classification results of classes and domains of the source image, the first target image, and the second target image.

9. The image processing apparatus according to claim 8 , further comprising:
a padder configured to increase the number of the second target images so that the number of the second target images acquired by the third image acquisitor is equal between classes.

10. The image processing apparatus according to claim 1 , wherein
the self-learner is configured to perform the self-learning by inputting, to the feature extractor, the first target image, as teacher data, in which a certainty factor of the pseudo label is equal to or greater than a threshold value among the first target images to each of which the pseudo label is assigned by the class classifier.

11. The image processing apparatus according to claim 1 , wherein
the class classifier and the domain classifier are configured to perform class classification and domain classification for each image group including a plurality of the source images and a plurality of the first target images, and
the self-learner is configured to determine whether to assign the pseudo label to the first target image every time one of the first target images in the image group is input to the second image acquisitor, and is configured to perform the self-learning based on the first tar

Assignees

• Kioxia Corp

Inventors

• Wang Bo
• Ng Youyang
• Lin Yuchieh
• Nakata Kengo
• Fujiwara Takeshi

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/40

  Extraction of image or video features · CPC title