Detail-preserving image editing techniques

Patent metadata

Field	Value
Publication number	US-11907839-B2
Application number	US-202117468511-A
Country	US
Kind code	B2
Filing date	Sep 7, 2021
Priority date	Oct 16, 2020
Publication date	Feb 20, 2024
Grant date	Feb 20, 2024

Abstract

Systems and methods combine an input image with an edited image generated using a generator neural network to preserve detail from the original image. A computing system provides an input image to a machine learning model to generate a latent space representation of the input image. The system provides the latent space representation to a generator neural network to generate a generated image. The system generates multiple scale representations of the input image, as well as multiple scale representations of the generated image. The system generates a first combined image based on first scale representations of the images and a first value. The system generates a second combined image based on second scale representations of the images and a second value. The system blends the first combined image with the second combined image to generate an output image.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
providing, by a computing system, an input image as input to a machine learning model to generate a latent space representation of the input image;
providing, by the computing system, the latent space representation of the input image as input to a trained generator neural network implemented by the computing system;
generating, by the generator neural network, a generated image based on the latent space representation of the input image;
generating, by the computing system, a first scale representation of the input image and a second scale representation of the input image;
generating, by the computing system, a first scale representation of the generated image and a second scale representation of the generated image;
generating, by the computing system, a first combined image based on the first scale representation of the input image, the first scale representation of the generated image, and a first value;
generating, by the computing system, a second combined image based on the second scale representation of the input image, the second scale representation of the generated image, and a second value different from the first value; and
blending, by the computing system, the first combined image with the second combined image to generate an output image.

2. The method of claim 1 , further comprising:
detecting, by the computing system, a first landmark in the input image;
detecting, by the computing system, a second landmark in the generated image, wherein the first landmark corresponds to the second landmark; and
warping, by the computing system, the input image to align the first landmark with the second landmark to produce a warped input image, wherein the warped input image is used to generate the first scale representation of the input image and the second scale representation of the input image.

3. The method of claim 1 , further comprising:
applying, by the computing system, edits to the latent space representation of the input image, wherein the edits are reflected in the output image.

4. The method of claim 1 , further comprising outputting, by the computing system, the output image for display.

5. The method of claim 1 , wherein generating the first scale representations of the images and the second scale representations of the images comprises calculating a Laplacian pyramid of the input image and a Laplacian pyramid of the generated image.

6. The method of claim 1 , further comprising:
generating, by the computing system, a third scale representation of the input image;
generating, by the computing system, a third scale representation of the generated image; and
generating, by the computing system, a third combined image based on the third scale representation of the input image, the third scale representation of the generated image, and a third value different from the first and second values,
wherein generating the output image further comprises blending the first combined image and the second combined image with the third combined image.

7. The method of claim 1 , wherein blending the first combined image with the second combined image to generate the output image comprises performing a Laplacian blending of the first combined image and the second combined image.

8. The method of claim 1 , further comprising masking, by the computing system, a region of the generated image.

9. A computing system comprising:
a processor;
a non-transitory computer-readable medium comprising instructions which, when executed by the processor, perform processing comprising:
obtaining a latent space representation of an input image;
providing the latent space representation of the input image as input to a trained generator neural network implemented by the computing system;
generating, by the generator neural network, a generated image based on the latent space representation of the input image;
generating a first scale representation of the input image and a second scale representation of the input image;
generating a first scale representation of the generated image and a second scale representation of the generated image;
generating a first combined image based on the first scale representation of the input image, the first scale representation of the generated image, and a first value;
generating a second combined image based on the second scale representation of the input image, the second scale representation of the generated image, and a second value different from the first value; and
blending the first combined image with the second combined image to generate an output image.

10. The computing system of claim 9 , the processing further comprising:
detecting, by the computing system, a first landmark in the input image;
detecting, by the computing system, a second landmark in the generated image, wherein the first landmark corresponds to the second landmark; and
warping, by the computing system, the input image to align the first landmark with the second landmark to produce a warped input image, wherein the warped input image is used to generate the first scale representation of the input image and the second scale representation of the input image.

11. The computing system of claim 9 , the processing further comprising:
applying, by the computing system, edits to the latent space representation of the input image, wherein the edits are reflected in the output image.

12. The computing system of claim 9 , the processing further comprising:
outputting the output image for display.

13. The computing system of claim 9 , wherein generating the first resolution images and the second resolution images comprises calculating a Laplacian pyramid of the input image and a Laplacian pyramid of the generated image.

14. The computing system of claim 9 , the processing further comprising:
generating, by the computing system, a third scale representation of the input image;
generating, by the computing system, a third scale representation of the generated image; and
generating, by the computing system, a third combined image based on the third scale representation of the input image, the third scale representation of the generated image, and a third value different from the first and second values,
wherein generating the output image further comprises blending the first combined image and the second combined image with the third combined image.

15. The computing system of claim 9 , wherein blending the first combined image with the second combined image to generate the output image comprises performing a Laplacian blending of the first combined image and the second combined image.

16. A non-transitory computer-readable medium having instructions stored thereon, the instructions executable by a processing device to perform operations comprising:
providing an input image as input to a machine learning model to generate a latent space representation of the input image;
providing the latent space representation of the input image as input to a trained generator neural network;
generating, by the generator neural network, a generated image based on the latent space representation of the input image; and
a step for producing an output image based on a first combined image of a first scale representation of the input image and a first scale representation of the generated image, and a second combined image of a second scale representation of the input image and a second scale representation of the generated image.

17. The medium of claim 16 , the operations further compr

Assignees

• Adobe Inc

Inventors

• Kalarot Ratheesh
• Wampler Kevin
• Lu Jingwan
• Fiser Jakub
• Shechtman Elya
• Darabi Aliakbar
• Costin Alexandru Vasile

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/094 Primary

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title