Segmentation guided image generation with adversarial networks

Patent metadata

Field	Value
Publication number	US-10825219-B2
Application number	US-201916361941-A
Country	US
Kind code	B2
Filing date	Mar 22, 2019
Priority date	Mar 22, 2018
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

Embodiments provide methods and systems for image generation through use of adversarial networks. An embodiment trains an image generator comprising (i) a generator implemented with a first neural network configured to generate a fake image based on a target segmentation, (ii) a discriminator implemented with a second neural network configured to distinguish a real image from a fake image and output a discrimination result as a function thereof and (iii) a segmentor implemented with a third neural network configured to generate a segmentation from the fake image. The training includes (i) operating the generator to output the fake image to the discriminator and the segmentor and (ii) iteratively operating the generator, discriminator, and segmentor during a training period, whereby the discriminator and generator train in an adversarial relationship with each other and the generator and segmentor train in a collaborative relationship with each other.

First claim

What is claimed is:

1. A system for training an image generator, the system comprising a processor and a memory with computer code instructions stored thereon, the processor and the memory, with the computer code instructions, being configured to cause the system to provide:
a generator implemented with a first neural network configured to generate a fake image based on a target segmentation;
a discriminator implemented with a second neural network configured to distinguish a real image from a fake image and output a discrimination result as a function thereof; and
a segmentor implemented with a third neural network configured to generate a segmentation from the fake image;
wherein the generator outputs the fake image to the discriminator and the segmentor; and
wherein iterative operation of the generator, discriminator, and segmentor during a training period causes the discriminator and generator to train in an adversarial relationship with each other and the generator and segmentor to train in a collaborative relationship with each other, the generator at the end of the training period having its first neural network trained to generate the fake image based on the target segmentation with more accuracy than at the start of the training period.

2. The system of claim 1 wherein the generator is further configured to generate the fake image based on the target segmentation and target attributes.

3. The system of claim 2 wherein the generator is further configured to generate the fake image based on the target segmentation, the target attributes, and a real image.

4. The system of claim 3 wherein a given fake image is a translated version of the real image.

5. The system of claim 2 wherein the generator is further configured to generate the fake image based on the target segmentation, the target attributes, and a latent vector.

6. The system of claim 5 wherein the latent vector is a random vector sampled from a normal distribution.

7. The system of claim 1 wherein, to implement the discriminator and generator to train in an adversarial relationship with each other:
the discriminator is configured to output the discrimination result to an optimizer; and
the optimizer is configured to: (i) adjust weights of the first neural network based on the discrimination result to improve generation of the fake image by the generator and (ii) adjust weights of the second neural network based on the discrimination result to improve distinguishing a real image from a fake image by the discriminator.

8. The system of claim 1 wherein the first neural network includes:
a down-sampling convolutional block configured to extract features of the target segmentation;
a first concatenation block configured to concatenate the extracted features with a latent vector;
an up-sampling block configured to construct a layout of the fake image using the concatenated extracted features and latent vector;
a second concatenation block configured to concatenate the layout with an attribute label to generate a multidimensional matrix representing features of the fake image; and
an up-sampling convolutional block configured to generate the fake image using the multidimensional matrix.

9. The system of claim 1 wherein the fake image is: an image of a person, an image of a vehicle, or an image of a person in clothes.

10. A method for training an image generator, the method comprising:
training: (i) a generator, implemented with a first neural network, to generate a fake image based on a target segmentation, (ii) a discriminator, implemented with a second neural network, to distinguish a real image from a fake image and output a discrimination result as a function thereof, and (iii) a segmentor, implemented with a third neural network, to generate a segmentation from the fake image, the training including:
by the generator, outputting the fake image to the discriminator and the segmentor; and
iteratively operating the generator, discriminator, and segmentor during a training period, the iterative operating causing the discriminator and generator to train in an adversarial relationship with each other and the generator and segmentor to train in a collaborative relationship with each other, the generator at the end of the training period having its first neural network trained to generate the fake image based on the target segmentation with more accuracy than at the start of the training period.

11. The method of claim 10 further comprising:
training the generator to generate the fake image based on the target segmentation and target attributes.

12. The method of claim 11 further comprising:
training the generator to generate the fake image based on the target segmentation, the target attributes, and a real image.

13. The method of claim 12 wherein a given fake image is a translated version of the real image.

14. The method of claim 11 further comprising:
training the generator to generate the fake image based on the target segmentation, the target attributes, and a latent vector.

15. The method of claim 14 wherein the latent vector is a random vector sampled from a normal distribution.

16. The method of claim 10 wherein causing the discriminator and generator to train in an adversarial relationship with each other includes:
by the discriminator, outputting the discrimination result to an optimizer; and
by the optimizer: (i) adjusting weights of the first neural network based on the discrimination result to improve generation of the fake image by the generator and (ii) adjusting weights of the second neural network based on the discrimination result to improve distinguishing a real image from a fake image by the discriminator.

17. The method of claim 10 wherein the generator, implemented with the first neural network, is trained to generate the fake image by:
at a down-sampling convolutional block, extracting features of the target segmentation;
at a first concatenation block, concatenating the extracted features with a latent vector;
at an up-sampling block, constructing a layout of the fake image using the concatenated extracted features and latent vector;
at a second concatenation block, concatenating the layout with an attribute label to generate a multidimensional matrix representing features of the fake image; and
at an up-sampling convolutional block, generating the fake image using the multidimensional matrix.

18. The method of claim 10 wherein the fake image is: an image of a person, an image of a vehicle, or an image of a person in clothes.

19. A computer program product for training an image generator, the computer program product comprising:
one or more non-transitory computer-readable storage devices and program instructions stored on at least one of the one or more storage devices, the program instructions, when loaded and executed by a processor, cause an apparatus associated with the processor to:
train: (i) a generator, implemented with a first neural network, to generate a fake image based on a target segmentation, (ii) a discriminator, implemented with a second neural network, to distinguish a real image from a fake image and output a discrimination result as a function thereof, and (iii) a segmentor, implemented with a third neural network, to generate a segmentation from the fake image, the training including:
by the generator, outputting the fake image to the discriminator and the segmentor; and
iteratively operating the generator, discriminator, and segmentor during a training period, the iterative operating

Assignees

• Univ Northeastern

Inventors

• Fu Yun
• Jiang Songyao

Classifications

• G06V40/16

  Human faces, e.g. facial parts, sketches or expressions · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title