Texture interpolation using neural networks

Patent metadata

Field	Value
Publication number	US-10818043-B1
Application number	US-201916392968-A
Country	US
Kind code	B1
Filing date	Apr 24, 2019
Priority date	Apr 24, 2019
Publication date	Oct 27, 2020
Grant date	Oct 27, 2020

Abstract

An example method for neural network based interpolation of image textures includes training a global encoder network to generate global latent vectors based on training texture images, and training a local encoder network to generate local latent tensors based on the training texture images. The example method further includes interpolating between the global latent vectors associated with each set of training images, and interpolating between the local latent tensors associated with each set of training images. The example method further includes training a decoder network to generate reconstructions of the training texture images and to generate an interpolated texture based on the interpolated global latent vectors and the interpolated local latent tensors. The training of the encoder and decoder networks is based on a minimization of a loss function of the reconstructions and a minimization of a loss function of the interpolated texture.

First claim

What is claimed is:

1. A method for interpolating textures, the method comprising:
applying, by a processor-based system, a first texture image to a global encoder network, the global encoder network trained to generate a first set of global latent vectors based on the first texture image;
applying, by the processor-based system, a second texture image to the global encoder network to generate a second set of global latent vectors based on the second texture image;
performing, by the processor-based system, a spatial repetition of the first set of global latent vectors and of the second set of global latent vectors;
interpolating, by the processor-based system, between the spatial repetition of the first set of global latent vectors and the spatial repetition of the second set of global latent vectors, to generate a global vector interpolation;
applying, by the processor-based system, the global vector interpolation to a decoder network, the decoder network trained to generate an interpolated texture based on the global vector interpolation.

2. The method of claim 1 , further comprising:
applying, by the processor-based system, the first texture image to a local encoder network, the local encoder network trained to generate a first set of local latent tensors based on the first texture image;
applying, by the processor-based system, the second texture image to the local encoder network to generate a second set of local latent tensors based on the second texture image;
performing, by the processor-based system, a spatial tiling operation on the first set of local latent tensors and on the second set of local latent tensors;
interpolating, by the processor-based system, between the tiled first set of local latent tensors and the tiled second set of local latent tensors, to generate a local tensor interpolation; and
applying, by the processor-based system, the local tensor interpolation to the decoder network, the decoder network trained to generate an interpolated texture based on the global vector interpolation and the local tensor interpolation.

3. The method of claim 2 , wherein the local encoder network comprises a first local encoder network to process a first set of regions of an applied texture image, and a second local encoder network to process a second set of regions of the applied texture image, the first set of regions associated with a first set of spatial dimensions, the second set of regions associated with a second set of spatial dimensions.

4. The method of claim 2 , wherein the global latent vector interpolation and the local latent tensor interpolation are based on linear weighting factors, the linear weighting factors being user provided or heuristically selected.

5. The method of claim 2 , wherein the training of the global encoder network, the local encoder network, and the decoder network is based on (1) a minimization of a reconstruction loss function of training texture images and (2) a minimization of an interpolation loss function of interpolated training texture images.

6. The method of claim 2 , wherein the spatial repetition comprises reshaping the first and second sets of global latent vectors into tensors and repeating the tensors to match spatial dimensions of the first and second sets of local latent tensors.

7. The method of claim 1 , wherein the decoder network is a generative adversarial network.

8. A method for training a neural network based texture interpolation system, the method comprising:
training, by a processor-based system, a global encoder network to generate a first set of global latent vectors based on a first training texture image and to generate a second set of global latent vectors based on a second training texture image;
training, by the processor-based system, a local encoder network to generate a first set of local latent tensors based on the first training texture image and to generate a second set of local latent tensors based on the second training texture image;
interpolating, by the processor-based system, between the first set and the second set of the global latent vectors;
interpolating, by the processor-based system, between the first set and the second set of the local latent tensors; and
training, by the processor-based system, a decoder network to generate a reconstruction of the first training texture image, and an interpolated texture based on the interpolated global latent vectors and the interpolated local latent tensors,
wherein the training of the global encoder network, the local encoder network, and the decoder network is based on a minimization of a loss function of the reconstruction of the first training texture image and a minimization of a loss function of the interpolated texture.

9. The method of claim 8 , wherein the local encoder network comprises a first local encoder network to process a first set of regions of a training texture image, and a second local encoder network to process a second set of regions of the training texture image, the first set of regions associated with a first set of spatial dimensions, the second set of regions associated with a second set of spatial dimensions.

10. The method of claim 9 , wherein the shuffle operation is a random shuffle of the tiled local latent tensors by row and column over a plurality of spatial scales.

11. The method of claim 8 , further comprising performing a spatial tiling operation on the first and second sets of local latent tensors and performing a shuffle operation on the tiled local latent tensors.

12. The method of claim 8 , wherein the loss function of the reconstructions comprises one or more of a pixel-wise sum of absolute differences loss, a Gram matrix loss, and an adversarial loss; and the loss function of the interpolated texture comprises a Gram matrix loss and an adversarial loss.

13. The method of claim 8 , wherein the global latent vector interpolation and the local latent tensor interpolation are based on linear weighting factors.

14. The method of claim 8 , wherein the decoder network is a generative adversarial network.

15. A computer program product including one or more non-transitory machine-readable mediums encoded with instructions that when executed by one or more processors cause a process to be carried out for texture interpolation, the process comprising:
applying a first texture image to a local encoder network, the local encoder network trained to generate a first set of local latent tensors based on the first texture image;
applying a second texture image to the local encoder network to generate a second set of local latent tensors based on the second texture image;
performing a tiling operation on the first set of local latent tensors and on the second set of local latent tensors;
interpolating between the tiled first set of local latent tensors and the tiled second set of local latent tensors, to generate a local tensor interpolation; and
applying the local tensor interpolation to a decoder network, the decoder network trained to generate an interpolated texture based on the local tensor interpolation.

16. The computer program product of claim 15 , the process further comprising:
applying the first texture image to a global encoder network, the global encoder network trained to generate a first set of global latent vectors based on the first texture image;
applying the second texture image to the global encoder network to generate a second set of global latent vectors based on the second texture image;
performing a spatial repetition of the first set of global latent vectors and of the second set of global la

Assignees

• Adobe Inc

Inventors

• Barnes Connelly
• Amirghodsi Sohrab
• Lukac Michal
• Shechtman Elya
• Yu Ning

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T7/40 Primary

  Analysis of texture (depth or shape recovery from texture G06T7/529) · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T11/001 Primary

  Physics · mapped topic