Performing patch matching guided by a transformation gaussian mixture model

Patent metadata

Field	Value
Publication number	US-11823313-B2
Application number	US-202117332773-A
Country	US
Kind code	B2
Filing date	May 27, 2021
Priority date	Feb 27, 2018
Publication date	Nov 21, 2023
Grant date	Nov 21, 2023

Abstract

The present disclosure is directed toward systems, methods, and non-transitory computer readable media for generating a modified digital image by identifying patch matches within a digital image utilizing a Gaussian mixture model. For example, the systems described herein can identify sample patches and corresponding matching portions within a digital image. The systems can also identify transformations between the sample patches and the corresponding matching portions. Based on the transformations, the systems can generate a Gaussian mixture model, and the systems can modify a digital image by replacing a target region with target matching portions identified in accordance with the Gaussian mixture model.

First claim

What is claimed is:

1. A computer-implemented method for patch matching to replace target regions of digital images comprising:
identifying a target region of a digital image to be modified;
sampling regions of the digital image to identify a target matching portion for the target region based on a transformation Gaussian mixture model, the transformation Gaussian mixture model defining probability distributions of transformation relationships between patches of the digital image, the transformation relationships comprising one or more of translation, rotation, scaling, or reflection; and
generate a modified digital image by replacing the target region with pixels from the target matching portion.

2. The computer-implemented method of claim 1 , wherein sampling regions of the digital image to identify a target matching portion for the target region based on a transformation Gaussian mixture model comprises sampling from areas of the digital image having a threshold probability of finding a matching portion as indicated by the transformation Gaussian mixture model.

3. The computer-implemented method of claim 1 , wherein the transformation relationships comprise at least two of: translation, rotation, scaling, or reflection.

4. The computer-implemented method of claim 1 , wherein the transformation relationships comprise a number of transformation types and each Gaussian distribution within a number of Gaussian distributions of the transformation Gaussian mixture model comprises a number of dimensions equal to the number of transformation types.

5. The computer-implemented method of claim 1 , further comprising:
receiving user input to modify a focus of Gaussian distributions of the transformation Gaussian mixture model; and
updating the transformation Gaussian mixture model by modifying the focus of the Gaussian distributions.

6. The computer-implemented method of claim 5 , further comprising:
resampling regions of the digital image to identify an updated target matching portion for the target region based on the updated transformation Gaussian mixture model; and
generate another modified digital image by replacing the target region with pixels from the updated target matching portion.

7. The computer-implemented method of claim 5 , wherein updating the transformation Gaussian mixture model by modifying the focus of the Gaussian distributions comprises modifying the Gaussian distributions to have tighter peaks.

8. A system for patch matching to replace target regions of digital images comprising:
one or more memory devices comprising:
a digital image; and
a transformation Gaussian mixture model that represents probability distributions of transformation relationships between patches of the digital image, the transformation relationships comprising one or more of translation, rotation, scaling, or reflection; and

at least one processor configured to cause the system to:
identify a target region of the digital image to be modified;
sample regions of the digital image to identify a target matching portion for the target region based on the transformation Gaussian mixture model; and
generate a modified digital image by replacing the target region with pixels from the target matching portion.

9. The system of claim 8 , wherein the at least one processor is further configured to cause the system to sample regions of the digital image to identify the target matching portion for the target region based on the transformation Gaussian mixture model by sampling from peaks of Gaussian distributions transformation Gaussian mixture model.

10. The system of claim 8 , wherein the at least one processor is further configured to cause the system to sample regions of the digital image to identify the target matching portion for the target region based on the transformation Gaussian mixture model by sampling from areas of the digital image having a threshold probability of finding a matching portion as indicated by the transformation Gaussian mixture model.

11. The system of claim 8 , wherein the at least one processor is further configured to cause the system to identify the target matching portion for the target region by comparing the target matching portion with pixels neighboring the target region.

12. The system of claim 8 , wherein:
comparing the target matching portion with the pixels neighboring the target region comprises generating a similarity score; and
wherein the at least one processor is further configured to cause the system to identify the target matching portion for the target region based on the similarity score being above a similarity threshold.

13. The system of claim 8 , wherein the at least one processor is further configured to cause the system to sample regions of the digital image to identify the target matching portion for the target region based on the transformation Gaussian mixture model by utilizing the transformation Gaussian mixture model to guide a correspondence search for target matching portions to replace the target region.

14. The system of claim 8 , wherein the transformation relationships between the patches of the digital image comprise at least two of the following: translation, rotation, scaling, or reflection.

15. The system of claim 8 , wherein the transformation Gaussian mixture model represents the probability distributions of the transformation relationships utilizing Gaussian distributions comprising a number of dimensions equal to a number of transformation types of the transformation relationships between the patches of the digital image.

16. A non-transitory computer readable medium for patch matching to replace target regions of digital images comprising instructions that, when executed by a processing device, cause to the processing device to perform operations comprising:
identifying a target region of a digital image to be modified;
utilizing a transformation Gaussian mixture model to identify a target matching portion from the digital image for the target region that is transformed by one or more of translation, rotation, scaling, or reflection relative to the target region, wherein the transformation Gaussian mixture model represents probability distributions of transformation relationships between patches of the digital image; and
generate a modified digital image by replacing the target region with pixels from the target matching portion.

17. The non-transitory computer readable medium of claim 16 , wherein identifying the target matching portion for the target region based on the transformation Gaussian mixture model comprises sampling from peaks of Gaussian distributions of the transformation Gaussian mixture model.

18. The non-transitory computer readable medium of claim 16 , wherein identifying the target matching portion for the target region based on the transformation Gaussian mixture model comprises sampling from areas of the digital image having a threshold probability of finding a matching portion as indicated by the transformation Gaussian mixture model.

19. The non-transitory computer readable medium of claim 16 , wherein identifying the target matching portion for the target region based on the transformation Gaussian mixture model comprises utilizing the transformation Gaussian mixture model to guide a correspondence search for target matching portions to replace the target region.

20. The non-transitory computer readable medium of claim 16 , wherein identifying the target matching portion for the target region based on the transformation Gaussi

Assignees

• Adobe Inc

Inventors

• Sun Xin
• Amirghodsi Sohrab
• Carr Nathan
• Lukac Michal

Classifications

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06V10/758

  Involving statistics of pixels or of feature values, e.g. histogram matching · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T5/77

  Retouching; Inpainting; Scratch removal · CPC title