Compressing image-to-image models with average smoothing

Patent metadata

Field	Value
Publication number	US-11790565-B2
Application number	US-202117191970-A
Country	US
Kind code	B2
Filing date	Mar 4, 2021
Priority date	Mar 4, 2021
Publication date	Oct 17, 2023
Grant date	Oct 17, 2023

Abstract

System and methods for compressing image-to-image models. Generative Adversarial Networks (GANs) have achieved success in generating high-fidelity images. An image compression system and method adds a novel variant to class-dependent parameters (CLADE), referred to as CLADE-Avg, which recovers the image quality without introducing extra computational cost. An extra layer of average smoothing is performed between the parameter and normalization layers. Compared to CLADE, this image compression system and method smooths abrupt boundaries, and introduces more possible values for the scaling and shift. In addition, the kernel size for the average smoothing can be selected as a hyperparameter, such as a 3×3 kernel size. This method does not introduce extra multiplications but only addition, and thus does not introduce much computational overhead, as the division can be absorbed into the parameters after training.

First claim

What is claimed is:

1. A method of operating a generative adversarial network (GAN), comprising:
receiving an image having learned parameters;
using an input class of the image to determine scaling and shifting parameters in a normalization layer; and
compressing the image using the determined scaling and shifting parameters by performing average smoothing between parameter layers and normalization layers to smooth abrupt boundaries where semantic information changes.

2. The method as specified in claim 1 wherein the learned parameters include spatial dependency.

3. The method as specified in claim 1 wherein the average smoothing generates a plurality of values for the scaling and shifting parameters.

4. The method as specified in claim 1 further comprising using an inception-based residual block containing a kernel.

5. The method as specified in claim 4 wherein the kernel has a kernel size selected from different kernel sizes.

6. The method as specified in claim 4 wherein the inception-based block incorporates depth-wise convolutional layers.

7. The method as specified in claim 1 , wherein the GAN is stored on a mobile computing device.

8. The method as specified in claim 1 , wherein the GAN is a pre-trained GAN.

9. A system comprising:
a processor; and
a memory storing computer readable instructions that, when executed by the processor, configure the system to perform operations comprising:
receiving an image having learned parameters;
using an input class of the image to determine scaling and shifting parameters in a normalization layer; and
compressing the image using the determined scaling and shifting parameters by performing average smoothing between parameter layers and normalization layers to smooth abrupt boundaries where semantic information changes.

10. The system as specified in claim 9 wherein the learned parameters include spatial dependency.

11. The system as specified in claim 9 wherein the average smoothing generates a plurality of values for the scaling and shifting parameters.

12. The system as specified in claim 9 further comprising using an inception-based residual block containing a kernel.

13. The system as specified in claim 12 wherein the kernel has a kernel size selected from different kernel sizes.

14. The system as specified in claim 12 wherein the inception-based block incorporates depth-wise convolutional layers.

15. The system as specified in claim 9 , wherein a generative adversial network (“GAN”) is stored on a mobile computing device.

16. The system as specified in claim 15 , wherein the GAN is a pre-trained GAN.

17. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:
receiving an image having learned parameters;
using an input class of the image to determine scaling and shifting parameters in a normalization layer; and
compressing the image using the determined scaling and shifting parameters by performing average smoothing between parameter layers and normalization layers to smooth abrupt boundaries where semantic information changes.

18. The computer-readable storage medium of claim 17 , wherein the learned parameters include spatial dependency.

19. The computer-readable storage medium of claim 17 , wherein the average smoothing generates a plurality of values for the scaling and shifting parameters.

20. The computer-readable storage medium of claim 17 , further comprising instructions to use an inception-based residual block containing a kernel.

Assignees

• Ren Jian
• Chai Menglei
• Tulyakov Sergey
• Jin Qing
• Snap Inc

Inventors

• Ren Jian
• Chai Menglei
• Tulyakov Sergey
• Jin Qing

Classifications

• G06N3/094

  Adversarial learning · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title