Adaptive convolutions in neural networks

Patent metadata

Field	Value
Publication number	US-12340440-B2
Application number	US-202117223577-A
Country	US
Kind code	B2
Filing date	Apr 6, 2021
Priority date	Nov 16, 2020
Publication date	Jun 24, 2025
Grant date	Jun 24, 2025

Abstract

A technique for performing style transfer between a content sample and a style sample is disclosed. The technique includes applying one or more neural network layers to a first latent representation of the style sample to generate one or more convolutional kernels. The technique also includes generating convolutional output by convolving a second latent representation of the content sample with the one or more convolutional kernels. The technique further includes applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes of the style sample.

First claim

What is claimed is:

1. A method for performing style transfer between a content sample and a style sample, comprising:
compressing a feature embedding generated from the style sample into a first latent representation comprising a d-dimensional vector;
applying one or more neural network layers to the first latent representation of the style sample to generate a first set of convolutional kernels associated with a first level of granularity of the style sample and a second set of convolutional kernels associated with a second level of granularity of the style sample;
generating convolutional output by convolving a second latent representation of the content sample with the first set of convolutional kernels; and
applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes associated with the first level of granularity of the style sample.

2. The method of claim 1 , further comprising updating a first set of weights in the one or more neural network layers and a second set of weights in the one or more decoder layers based on one or more losses calculated between the style transfer result and at least one of the content sample or the style sample.

3. The method of claim 2 , wherein the one or more losses comprise a style loss between a third latent representation of the style transfer result and the first latent representation of the style sample.

4. The method of claim 2 , wherein the one or more losses comprise a content loss between a third latent representation of the style transfer result and the second latent representation of the content sample.

5. The method of claim 2 , wherein the one or more losses comprise a weighted sum of a first loss between the style transfer result and the style sample and a second loss between the style transfer result and the content sample.

6. The method of claim 1 , further comprising applying an encoder network to the content sample to produce the second latent representation as a feature embedding of the content sample.

7. The method of claim 1 , further comprising generating, as additional output of the one or more neural network layers, one or more biases to be applied after the one or more convolutional kernels.

8. The method of claim 1 , further comprising:
applying an encoder network to the style sample to produce the feature embedding of the style sample; and
inputting the feature embedding into one or more additional neural network layers to produce the d-dimensional vector.

9. The method of claim 1 , wherein generating the convolutional output comprises:
convolving the second latent representation with a first kernel in the first set to produce a first output matrix at a first resolution;
applying one or more additional neural network layers to the first output matrix to produce a modified output matrix; and
convolving the modified output matrix with a second kernel in the second set to produce a second output matrix at a second resolution that is higher than the first resolution.

10. The method of claim 1 , wherein at least a portion of the convolutional output is generated using the one or more decoder layers.

11. The method of claim 1 , wherein the content sample and the style sample comprise at least one of an image or a mesh.

12. The method of claim 1 , wherein the one or more convolutional kernels comprise at least one of a depthwise convolution, a pointwise convolution, or a per-channel bias.

13. The method of claim 1 , wherein the content sample comprises a representation of a scene and the style sample comprises one or more parameters that control a depiction of the scene.

14. The method of claim 13 , wherein the one or more parameters comprise at least one of a lighting parameter or a camera parameter.

15. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of:
compressing a feature embedding generated from the style sample into a first latent representation comprising a d-dimensional vector;
applying one or more neural network layers to the first latent representation of the style sample to generate a first set of convolutional kernels associated with a first level of granularity of the style sample and a second set of convolutional kernels associated with a second level of granularity of the style sample;
generating convolutional output by convolving a second latent representation of the content sample with the first set of convolutional kernels; and
applying one or more decoder layers to the convolutional output to produce a style transfer result that comprises one or more content-based attributes of the content sample and one or more style-based attributes associated with the first level of granularity of the style sample.

16. The non-transitory computer readable medium of claim 15 , wherein, when executed by the processor, the instructions further cause the processor to perform the steps of updating a first set of weights in the one or more neural network layers and a second set of weights in the one or more decoder layers based on one or more losses calculated between the style transfer result and at least one of the content sample or the style sample.

17. The non-transitory computer readable medium of claim 16 , wherein the one or more losses comprise a weighted sum of a style loss between a third latent representation of the style transfer result and the first latent representation of the style sample and a content loss between the third latent representation of the style transfer result and the second latent representation of the content sample.

18. The non-transitory computer readable medium of claim 15 , wherein, when executed by the processor, the instructions further cause the processor to perform the steps of:
applying an encoder network to the style sample to produce a first feature embedding of the style sample; and
inputting the first feature embedding into one or more additional neural network layers to produce the d-dimensional vector.

19. The non-transitory computer readable medium of claim 18 , wherein, when executed by the processor, the instructions further cause the processor to perform the steps of:
applying the encoder network to the content sample to produce the second latent representation as a second feature embedding of the content sample; and
normalizing the second latent representation prior to generating the convolutional output.

20. The non-transitory computer readable medium of claim 15 , wherein generating the convolutional output comprises:
convolving the second latent representation with a first kernel in the first set to produce a first output matrix at a first resolution;
applying one or more additional neural network layers to the first output matrix to produce a modified output matrix; and
convolving the modified output matrix with a second kernel in the second set to produce a second output matrix at a second resolution that is higher than the first resolution.

21. The non-transitory computer readable medium of claim 15 , wherein the one or more content-based attributes comprise a recognizable arrangement of abstract shapes representing an object in the content sample.

22. The non-transitory computer readable medium of claim 15 , wherein the one or more style-based attributes comprise at least one of a line, an edge, a brush

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Chandran Prashanth
• Bradley Derek Edward
• Urnau Gotardo Paulo Fabiano
• Zoss Gaspard

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/04 Primary

  Architecture, e.g. interconnection topology · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title