Generation using depth-conditioned autoencoder

Patent metadata

Field	Value
Publication number	US-12437427-B2
Application number	US-202217896574-A
Country	US
Kind code	B2
Filing date	Aug 26, 2022
Priority date	Aug 26, 2022
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

An image processing system uses a depth-conditioned autoencoder to generate a modified image from an input image such that the modified image maintains an overall structure from the input image while modifying textural features. An encoder of the depth-conditioned autoencoder extracts a structure latent code from an input image and depth information for the input image. A generator of the depth-conditioned autoencoder generates a modified image using the structure latent code and a texture latent code. The modified image generated by the depth-conditioned autoencoder includes the structural features from the input image while incorporating textural features of the texture latent code. In some aspects, the autoencoder is depth-conditioned during training by augmenting training images with depth information. The autoencoder is trained to preserve the depth information when generating images.

First claim

What is claimed is:

1. One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising:
extracting, by an encoder of an autoencoder, a structure latent code from a first image and depth information for the first image, the structure latent code encoding local structural features of the first image;
extracting, by the encoder of the autoencoder, a texture latent code from a second image and depth information for the second image, the texture latent code encoding global textural features of the second image; and
generating, by a generator of the autoencoder, a third image using the structure latent code and the texture latent code.

2. The one or more computer storage media of claim 1 , wherein the operations further comprising:
generating the depth information for the first image by using a depth estimation model to predict depth values for pixels from the first image; and
generating the depth information for the second image by using the depth estimation model to predict depth values for pixels from the second image.

3. The one or more computer storage media of claim 1 , wherein the depth information for the first image comprises a first depth image having depth values for pixels of the first depth image, the depth values of the first depth image predicted by a depth estimation model using the first image, and wherein the depth information for the second image comprises a second depth image having depth values for pixels of the second depth image, the depth values predicted by the depth estimation model using the second image.

4. The one or more computer storage media of claim 3 , wherein the first depth image and the second depth image are blurred.

5. The one or more computer storage media of claim 1 , wherein the structure latent code is modified to provide a modified structure latent code prior to generating the third image, and wherein generating the third image using the structure latent code comprises generating the third image using the modified structure latent code.

6. The one or more computer storage media of claim 1 , wherein the texture latent code is modified to provide a modified texture latent code prior to generating the third image, and wherein generating the third image using the texture latent code comprises generating the third image using the modified texture latent code.

7. A computer system comprising:
a processor; and
a computer storage medium storing computer-useable instructions that, when used by the processor, causes the computer system to perform operations comprising:
receiving an image dataset of training images and depth information for at least a portion of the training images; and
training an autoencoder using the training images and depth information to generate a trained autoencoder including an encoder and a generator, the encoder mapping an input image and depth information for the input image to a structure latent code, the generator mapping the structure latent code for the input image and a texture latent code to an output image, wherein training the autoencoder includes an iteration that comprises:
extracting, using the encoder, a first structure latent code and a first texture latent code from a first training image and first depth information for the first training image,
extracting, using the encoder, a second structure latent code and a second texture latent code from a second training image and second depth information for the second training image,
generating, using the generator, a reconstructed image from the first structure latent code and the first texture latent code,
generating, using the generator, a modified image from the first structure latent code and the second texture latent code,
determining one or more losses based on the reconstructed image and the modified image, and
updating parameters of the autoencoder based on the one or more losses.

8. The computer system of claim 7 , wherein the one or more losses comprise a reconstruction loss based on a difference between the first training image with the first depth information and the reconstructed image.

9. The computer system of claim 7 , wherein the one or more losses comprise a reconstruction generative adversarial network (GAN) loss associated with a discriminator based on the reconstructed image.

10. The computer system of claim 7 , wherein the one or more losses comprise a swapping generative adversarial network (GAN) loss associated with a discriminator based on the modified image.

11. The computer system of claim 7 , wherein the one or more losses comprise a co-occurrence loss associated with a discriminator based on patches from the second training image and the modified image.

12. A computer-implemented method comprising:
extracting, by an encoder of an autoencoder, a structure latent code from a first image and depth information for the first image, the structure latent code encoding local structural features of the first image;
extracting, by the encoder of the autoencoder, a texture latent code from a second image and depth information for the second image, the texture latent code encoding global textural features of the second image; and
generating, by a generator of the autoencoder, a third image using the structure latent code and the texture latent code.

13. The computer-implemented method of claim 12 , wherein the operations further comprising:
generating the depth information for the first image by using a depth estimation model to predict depth values for pixels from the first image; and
generating the depth information for the second image by using the depth estimation model to predict depth values for pixels from the second image.

14. The computer-implemented method of claim 12 , wherein the depth information for the first image comprises a first depth image having depth values for pixels of the first depth image, the depth values of the first depth image predicted by a depth estimation model using the first image, and wherein the depth information for the second image comprises a second depth image having depth values for pixels of the second depth image, the depth values predicted by the depth estimation model using the second image.

15. The computer-implemented method of claim 14 , wherein the first depth image and the second depth image are blurred.

16. The computer-implemented method of claim 12 , wherein the structure latent code is modified to provide a modified structure latent code prior to generating the third image, and wherein generating the third image using the structure latent code comprises generating the third image using the modified structure latent code.

17. The computer-implemented method of claim 12 , wherein the texture latent code is modified to provide a modified texture latent code prior to generating the third image, and wherein generating the third image using the texture latent code comprises generating the third image using the modified texture latent code.

Assignees

• Adobe Inc

Inventors

• Park Taesung
• Paris Sylvain
• Zhang Richard
• Shechtman Elya

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/40 Primary

  Analysis of texture (depth or shape recovery from texture G06T7/529) · CPC title