Cyclic generative adversarial network for unsupervised cross-domain image generation

Patent metadata

Field	Value
Publication number	US-10474929-B2
Application number	US-201815906710-A
Country	US
Kind code	B2
Filing date	Feb 27, 2018
Priority date	Apr 25, 2017
Publication date	Nov 12, 2019
Grant date	Nov 12, 2019

Abstract

A system is provided for unsupervised cross-domain image generation relative to a first and second image domain that each include real images. A first generator generates synthetic images similar to real images in the second domain while including a semantic content of real images in the first domain. A second generator generates synthetic images similar to real images in the first domain while including a semantic content of real images in the second domain. A first discriminator discriminates real images in the first domain against synthetic images generated by the second generator. A second discriminator discriminates real images in the second domain against synthetic images generated by the first generator. The discriminators and generators are deep neural networks and respectively form a generative network and a discriminative network in a cyclic GAN framework configured to increase an error rate of the discriminative network to improve synthetic image quality.

First claim

What is claimed is:

1. A system for unsupervised cross-domain image generation relative to a first image domain and a second image domain that each include real images, comprising:
a first image generator for generating synthetic images having a similar appearance to one or more of the real images in the second image domain while including a semantic content of one or more of the real images in the first image domain;
a second image generator for generating synthetic images having a similar appearance to at least one of the real images in the first image domain while including a semantic content of at least one of the real images in the second image domain;
a first discriminator for discriminating the real images in the first image domain against the synthetic images generated by the second image generator; and
a second discriminator for discriminating the real images in the second image domain against the synthetic images generated by the first image generator;
wherein the discriminators and the generators are deep neural networks and respectively form a generative network and a discriminative network in a cyclic Generative Adversarial Network (GAN) framework that is configured to increase an error rate of the discriminative network to improve a quality of the synthetic images.

2. The system of claim 1 , wherein the cyclic GAN framework employs a cyclic consistency loss in order to preserve the semantic contents for inclusion in the generated synthetic images.

3. The system of claim 1 , wherein the first image domain and the second image domain include at least some different real images relative to each other.

4. The system of claim 1 , wherein the generators are implemented by respective convolutional neural networks, and the discriminators are implemented by respective de-convolutional neural networks.

5. The system of claim 1 , wherein the generators generate the synthetic images using gradients provided by the discriminators.

6. The system of claim 1 , wherein the generative network is configured to train another supervised learning element in an object category detection network.

7. The system of claim 1 , wherein the cyclic GAN framework is configured to perform a cyclic domain transfer with respect to the first image domain and the second image domain.

8. The system of claim 7 , wherein the cyclic GAN framework is configured to enforce cyclic consistency across the cyclic domain transfer while adapting the image properties from one of the domains to another one of the domains.

9. The system of claim 1 , wherein the generative network of the cyclic GAN framework is configured to generate the synthetic images for different weather conditions from the real images in any of the first domain and the second domain.

10. The system of claim 1 , wherein the cyclic GAN framework forms an unsupervised domain-to-domain translation model configured for unsupervised learning from a training dataset from among the domains.

11. A computer-implemented method for unsupervised cross-domain image generation relative to a first image domain and a second image domain that each include real images, comprising:
generating, by a first image generator, synthetic images having a similar appearance to one or more of the real images in the second image domain while including a semantic content of one or more of the real images in the first image domain;
generating, by a second image generator, synthetic images having a similar appearance to at least one of the real images in the first image domain while including a semantic content of at least one of the real images in the second image domain;
discriminating, by a first discriminator, the real images in the first image domain against the synthetic images generated by the second image generator; and
discriminating, by a second discriminator, the real images in the second image domain against the synthetic images generated by the first image generator,
wherein the discriminators and the generators are each neural network based and respectively form a generative network and a discriminative network in a cyclic Generative Adversarial Network (GAN) framework, and the method further comprises increasing an error rate of the discriminative network to improve a quality of the synthetic images.

12. The computer-implemented method of claim 11 , further comprising utilizing a cyclic consistency loss in the cyclic GAN framework in order to preserve the semantic contents for inclusion in the generated synthetic images.

13. The computer-implemented method of claim 11 , wherein the first image domain and the second image domain include at least some different real images relative to each other.

14. The computer-implemented method of claim 11 , further comprising:
configuring each of a pair of convolutional neural networks as a respective one of the generators; and
configuring each of a pair of de-convolutional neural networks as a respective one of the discriminators.

15. The computer-implemented method of claim 11 , wherein said generating steps generate the synthetic images using gradients provided by the discriminators.

16. The computer-implemented method of claim 11 , further comprising training, by the generative network, another supervised learning element in an object category detection network.

17. The computer-implemented method of claim 11 , further comprising performing, by the cyclic GAN framework, a cyclic domain transfer with respect to the first image domain and the second image domain.

18. The computer-implemented method of claim 17 , further comprising configuring the cyclic GAN framework to enforce cyclic consistency across the cyclic domain transfer while adapting the image properties from one of the domains to another one of the domains.

19. The computer-implemented method of claim 11 , further comprising configuring the generative network of the cyclic GAN framework to generate the synthetic images for different weather conditions from the real images in any of the first domain and the second domain.

20. A computer program product for unsupervised cross-domain image generation relative to a first image domain and a second image domain that each include real images, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
generating, by a first image generator of the computer, synthetic images having a similar appearance to one or more of the real images in the second image domain while including a semantic content of one or more of the real images in the first image domain;
generating, by a second image generator of the computer, synthetic images having a similar appearance to at least one of the real images in the first image domain while including a semantic content of at least one of the real images in the second image domain;
discriminating, by a first discriminator of the computer, the real images in the first image domain against the synthetic images generated by the second image generator; and
discriminating, by a second discriminator of the computer, the real images in the second image domain against the synthetic images generated by the first image generator,
wherein the discriminators and the generators are each neural network based and respectively form a generative network and a discriminative network in a cyclic Generative Adversarial Network (GAN) framework, and the method furth

Assignees

• Nec Lab America Inc
• Nec Corp

Inventors

• Choi Wongun
• Schulter Samuel
• Sohn Kihyuk
• Chandraker Manmohan

Classifications

• G06V10/7753

  Incorporation of unlabelled data, e.g. multiple instance learning [MIL] · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title