Latent code for unsupervised domain adaptation

Patent metadata

Field	Value
Publication number	US-11494660-B2
Application number	US-202016860727-A
Country	US
Kind code	B2
Filing date	Apr 28, 2020
Priority date	Jul 17, 2019
Publication date	Nov 8, 2022
Grant date	Nov 8, 2022

Abstract

A pre-trained source encoder generates a source encoder representation for each image of a labeled set of source images from a source domain. A target encoder generates a target encoder representation for each image of an unlabeled set of target images from a target domain. A generative adversarial network outputs a first prediction indicating whether each of the source encoder representations and each of the target encoder representations originate from the source domain or the target domain. The generative adversarial network outputs a second prediction of the latent code for each of the source encoder representations and each of the target encoder representations. The target encoder and the generative adversarial network are trained by repeatedly updating parameters of the target encoder and the generative adversarial network until one or more predetermined stopping conditions occur.

First claim

What is claimed is:

1. A computer-implemented method of training a target encoder for classifying images into one of a plurality of categories using a classifier trained on images from a source domain, the method comprising:
generating, by a pre-trained source encoder, a source encoder representation for each image of a labeled set of source images from the source domain;
generating by the target encoder, a target encoder representation for each image of an unlabeled set of target images from a target domain;
inputting, into a generative adversarial network, the source encoder representations, the target encoder representations, and latent code representing the plurality of categories;
by the generative adversarial network, outputting a first prediction indicating whether each of the source encoder representations and each of the target encoder representations originate from the source domain or the target domain;
by the generative adversarial network, outputting a second prediction of the latent code for each of the source encoder representations and each of the target encoder representations; and
training the target encoder and the generative adversarial network by repeatedly updating parameters of the target encoder and the generative adversarial network until one or more predetermined stopping conditions occur.

2. The method of claim 1 , wherein the training includes training the target encoder and the generative adversarial network by repeatedly updating parameters of the target encoder and the generative adversarial network until a first loss function for the first prediction reaches a minimum value and a second loss function for the second prediction reaches a minimum value.

3. The method claim 1 , further comprising:
classifying a target image using the trained target encoder and the classifier, wherein the classifier is trained using supervised learning on the labeled set of source images.

4. The method of claim 3 , wherein classifying a target image using the trained target encoder and the classifier, comprises:
inputting a target image into the trained target encoder;
generating a target encoder representation of the target image;
inputting the target encoder representation of the target image into the trained classifier; and
classifying the target image by the trained classifier into one of the plurality of categories.

5. The method of claim 1 , further comprising pre-training the source encoder and the classifier by:
inputting the source encoder representations and source labels into the classifier; and
pre-training the source encoder and the classifier by repeatedly updating parameters of the source encoder and the classifier until one or more second predetermined stopping conditions occur.

6. The method of claim 5 , wherein the pre-training includes pre-training the source encoder and the classifier by repeatedly updating parameters of the source encoder and the classifier until a classification loss function reaches a minimum value.

7. The method of claim 5 , wherein the parameters of the pre-trained source encoder are not fixed after the pre-training, and
wherein the method further comprises training the pre-trained source encoder jointly with the target encoder and the generative adversarial network by repeatedly updating parameters of the source encoder, the target encoder, and the generative adversarial network (A) until a classification loss function reaches a minimum value, a first loss function for the first prediction reaches a minimum value and a second loss function for the second prediction reaches a minimum value.

8. The method of claim 5 , wherein the parameters of the pre-trained source encoder are fixed after the pre-training.

9. The method of claim 1 , wherein the parameters of the pre-trained source encoder are fixed after the pre-training.

10. The method of claim 1 , wherein the parameters of the target encoder are initialized using parameters of the pre-trained source encoder.

11. The method of claim 1 , wherein the generative adversarial network comprises a generator neural network, a discriminator neural network, and an auxiliary neural network, and
wherein the inputting, into the generative adversarial network, the source encoder representations, the target encoder representations, and the latent code comprises inputting, into the generator neural network, the source encoder representations combined with the latent code and the target encoder representations combined with the latent code.

12. The method of claim 11 , further comprising:
computing, by the generator neural network, joint hidden representations for each of the source encoder representations and each of the target encoder representations, each joint hidden representation based on a combination of the latent code and a respective one of the source encoder representations and the target encoder representations;
inputting, into the discriminator neural network, the joint hidden representations, wherein the discriminator neural network is configured to output the first prediction; and
inputting, into the auxiliary neural network, the joint hidden representations, wherein the auxiliary neural network is configured to output the second prediction.

13. The method of claim 12 wherein outputting, by the auxiliary neural network, the second prediction of the latent code for each of the joint hidden representations comprises outputting a probability distribution for the latent code given the respective joint representation.

14. The method of claim 13 , wherein the auxiliary neural network includes a convolutional neural network comprising a final fully connected layer and a softmax function to output parameters for the probability distribution.

15. The method of claim 11 , wherein the generator neural network includes a convolutional neural network comprising three hidden layers with rectified linear unit (ReLu) activation.

16. The method of claim 11 , wherein the discriminator neural network includes a convolutional neural network comprising at least one fully connected layer.

17. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by one or more processors perform a method of training a target encoder for classifying images into one of a plurality of categories using a classifier trained on images from a source domain, the method comprising:
generating, by a pre-trained source encoder, a source encoder representation for each image of a labeled set of source images from the source domain;
generating, by the target encoder, a target encoder representation for each image of an unlabeled set of target images from a target domain;
inputting, into a generative adversarial network, the source encoder representations, the target encoder representations, and latent code representing the plurality of categories,
wherein the generative adversarial network is configured to:
output a first prediction indicating whether each of the source encoder representations and each of the target encoder representations originate from the source domain or the target domain; and
output a second prediction of the latent code for each of the source encoder representations and each of the target encoder representations; and

training the target encoder and the generative adversarial network by repeatedly updating parameters of the target encoder and the generative adversarial network until one or more predetermined stopping conditions occur.

18. A training system comprising:
a target encoder

Assignees

• Naver Corp

Inventors

• Chidlovskii Boris

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06F18/2413

  based on distances to training or reference patterns · CPC title

• G06F18/24

  Classification techniques · CPC title