Techniques for smooth region merging in image editing

Patent metadata

Field	Value
Publication number	US-11915133-B2
Application number	US-202117468546-A
Country	US
Kind code	B2
Filing date	Sep 7, 2021
Priority date	Oct 16, 2020
Publication date	Feb 27, 2024
Grant date	Feb 27, 2024

Abstract

Systems and methods seamlessly blend edited and unedited regions of an image. A computing system crops an input image around a region to be edited. The system applies an affine transformation to rotate the cropped input image. The system provides the rotated cropped input image as input to a machine learning model to generate a latent space representation of the rotated cropped input image. The system edits the latent space representation and provides the edited latent space representation to a generator neural network to generate a generated edited image. The system applies an inverse affine transformation to rotate the generated edited image and aligns an identified segment of the rotated generated edited image with an identified corresponding segment of the input image to produce an aligned rotated generated edited image. The system blends the aligned rotated generated edited image with the input image to generate an edited output image.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
cropping, by a computing system, an input image around a region to be edited to produce a cropped input image;
applying, by the computing system, an affine transformation to the cropped input image to produce a rotated cropped input image;
providing, by the computing system, the rotated cropped input image as input to a machine learning model to generate a latent space representation of the rotated cropped input image;
editing, by the computing system, the latent space representation to generate an edited latent space representation;
providing, by the computing system, the edited latent space representation as input to a trained generator neural network implemented by the computing system;
generating, by the generator neural network, a generated edited image;
applying, by the computing system, an inverse affine transformation to the generated edited image to generate a rotated generated edited image;

aligning, by the computing system, an identified segment of the rotated generated edited image with an identified corresponding segment of the input image to produce an aligned rotated generated edited image; and
blending, by the computing system, the aligned rotated generated edited image with the input image to generate an edited output image.

2. The method of claim 1 , further comprising:
identifying, by the computing system, the segment of the rotated generated edited image; and
identifying, by the computing system, the corresponding segment of the input image.

3. The method of claim 2 , wherein identifying the corresponding segment of the input image comprises:
providing, by the computing system, the input image to a segmentation neural network implemented by the computing system, wherein the segmentation neural network identifies a plurality of segments including the segment of the input image.

4. The method of claim 1 , further comprising:
based on the identified corresponding segment of the input image and the identified segment of the rotated generated edited image, modifying, by the computing system, a color and a contrast in the rotated generated edited image to match a color and a contrast in the input image.

5. The method of claim 1 , wherein blending the aligned rotated generated edited image with the input image comprises applying, by the computing system, healing to the aligned rotated generated edited image and the input image.

6. The method of claim 1 , further comprising:
identifying, by the computing system, pixels corresponding to artifacts in the aligned rotated generated edited image; and
applying a content-aware fill to the identified pixels of the aligned rotated generated edited image.

7. The method of claim 1 , further comprising outputting, by the computing system, the edited output image to a display device for display.

8. The method of claim 1 , further comprising, before cropping the input image, detecting, by the computing system, a target region of the input image for configuring the cropping.

9. A computing system comprising:
a processor;
a non-transitory computer-readable medium comprising instructions which, when executed by the processor, perform processing comprising:
applying an affine transformation to an input image to produce a rotated input image;
providing the rotated input image as input to a machine learning model to generate a latent space representation of the rotated input image;
editing the latent space representation to generate an edited latent space representation;
providing the edited latent space representation as input to a trained generator neural network implemented by the computing system;
generating, by the generator neural network, an edited generated image;
applying an inverse affine transformation to the generated edited image to generate a rotated generated edited image;
aligning an identified segment of the rotated generated edited image with an identified corresponding segment of the input image to produce an aligned rotated generated edited image; and
blending the aligned rotated generated edited image with the input image to generate an edited output image.

10. The computing system of claim 9 , the processing further comprising:
identifying the segment of the rotated generated edited image; and
identifying the corresponding segment of the input image.

11. The computing system of claim 10 , wherein identifying the corresponding segment of the input image comprises:
providing, by the computing system, the input image to a segmentation neural network implemented by the computing system, wherein the segmentation neural network identifies a plurality of segments including the segment of the input image.

12. The computing system of claim 9 , the processing further comprising:
outputting the edited output image for display.

13. The computing system of claim 9 , the processing further comprising:
based on the identified corresponding segment of the input image and the identified segment of the rotated generated edited image, adjusting, by the computing system, a color and a contrast in the rotated generated edited image to match a color and a contrast in the input image.

14. The computing system of claim 9 , wherein blending the aligned rotated generated edited image with the input image comprises applying, by the computing system, healing to the aligned rotated generated edited image and the input image.

15. The computing system of claim 9 , the processing further comprising:
identifying, by the computing system, pixels corresponding to artifacts in the aligned rotated generated image; and
applying a content-aware fill to the identified pixels of the aligned rotated generated image.

16. A non-transitory computer-readable medium having instructions stored thereon, the instructions executable by a processing device to perform operations comprising:
cropping an input image around a region to be edited to produce a cropped input image;
applying an affine transformation to the cropped input image to produce a rotated cropped input image;
providing the rotated cropped input image as input to a machine learning model to generate a latent space representation of the rotated cropped input image;
editing the latent space representation to generate an edited latent space representation;
providing the edited latent space representation as input to a trained generator neural network;
generating, by the generator neural network, a generated edited image; and
a step for blending the generated edited image with the input image such that an identified segment of the generated edited image aligns with an identified corresponding segment of the input image.

17. The medium of claim 16 , the operations further comprising:
identifying the segment of the generated edited image; and
identifying the corresponding segment of the input image.

18. The medium of claim 17 , wherein identifying the corresponding segment of the input image comprises:
providing the input image to a segmentation neural network, wherein the segmentation neural network identifies a plurality of segments including the segment of the input image.

19. The medium of claim 16 , the operations further comprising:
outputting the edited output image for display.

20. The medium of claim 16 , the operations further comprising:
identifying, by the computing system, pixels corresponding to artifacts in the aligned rotated generated image; and
applying a content-aware fill to

Assignees

• Adobe Inc

Inventors

• Kalarot Ratheesh
• Wampler Kevin
• Lu Jingwan
• Fiser Jakub
• Shechtman Elya
• Darabi Aliakbar
• Costin Alexandru Vasile

Classifications

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/094 Primary

  Adversarial learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title