Generating modified digital images utilizing nearest neighbor fields from patch matching operations of alternate digital images

Patent metadata

Field	Value
Publication number	US-12373915-B2
Application number	US-202217820649-A
Country	US
Kind code	B2
Filing date	Aug 18, 2022
Priority date	Nov 8, 2019
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for generating modified digital images by utilizing a patch match algorithm to generate nearest neighbor fields for a second digital image based on a nearest neighbor field associated with a first digital image. For example, the disclosed systems can identify a nearest neighbor field associated with a first digital image of a first resolution. Based on the nearest neighbor field of the first digital image, the disclosed systems can utilize a patch match algorithm to generate a nearest neighbor field for a second digital image of a second resolution larger than the first resolution. The disclosed systems can further generate a modified digital image by filling a target region of the second digital image utilizing the generated nearest neighbor field.

First claim

What is claimed is:

1. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computing device to:
perform a patch match algorithm on a digital image in a first resolution;
determine results from the patch match algorithm on the digital image in the first resolution; and
generate an inpainted digital image in a second resolution larger than the first resolution utilizing the patch match algorithm according to the results of the first resolution.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the results from the patch match algorithm in the first resolution by determining a first set of pixel offsets in the first resolution.

3. The non-transitory computer readable medium of claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine a second set of pixel offsets in the second resolution from the first set of pixel offsets in the first resolution.

4. The non-transitory computer readable medium of claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the inpainted digital image by filling pixels of a second digital image in the second resolution utilizing the patch match algorithm according to the second set of pixel offsets.

5. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the inpainted digital image by filling pixels of a panoramic digital image in the second resolution utilizing the patch match algorithm.

6. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to select a target scale for the patch match algorithm from a hierarchy of scales.

7. The non-transitory computer readable medium of claim 6 , further comprising instructions that, when executed by the at least one processor, cause the computing device to select the target scale by identifying a scale within the hierarchy of scales that is closest to and larger than the first resolution.

8. A system comprising:
one or more memory devices comprising a digital image and a patch match algorithm; and
one or more processors configured to cause the system to:
implement the patch match algorithm on a digital image in a first resolution;
determine, from implementing the patch match algorithm on the digital image, a first set of pixel offsets in the first resolution;
determine, from the first set of pixel offsets in the first resolution, a second set of pixel offsets in a second resolution larger than the first resolution; and
generate an inpainted digital image in the second resolution utilizing the patch match algorithm according to the second set of pixel offsets.

9. The system of claim 8 , wherein the one or more processors are further configured to cause the system to determine the first set of pixel offsets in the first resolution by determining a first nearest neighbor field associated with the digital image utilizing the patch match algorithm.

10. The system of claim 9 , wherein the one or more processors are further configured to cause the system to determine the second set of pixel offsets in the second resolution by determining a second nearest neighbor field associated with a second digital image in the second resolution utilizing the patch match algorithm in the second resolution.

11. The system of claim 10 , wherein the one or more processors are further configured to cause the system to determine the second set of pixel offsets in the second resolution by initializing the patch match algorithm at a target scale corresponding to the first resolution to avoid determining pixel offsets at resolutions smaller than the first resolution.

12. The system of claim 8 , wherein the one or more processors are further configured to cause the system to generate the inpainted digital image by generating a panoramic digital image in the second resolution utilizing the second set of pixel offsets to stitch two or more digital images together.

13. The system of claim 12 , wherein the one or more processors are further configured to cause the system to determine the first set of pixel offsets from a thumbnail version of the panoramic digital image utilizing the patch match algorithm.

14. A computer-implemented method comprising:
performing a patch match algorithm on a digital image in a first resolution;
determining results from the patch match algorithm on the digital image in the first resolution;
generating an inpainted digital image in a second resolution larger than the first resolution utilizing the patch match algorithm according to the results of the first resolution; and
providing the inpainted digital image for display on a client device.

15. The computer-implemented method of claim 14 , wherein determining the results from the patch match algorithm comprises determining a first nearest neighbor field defining a first set of pixel offsets in the first resolution.

16. The computer-implemented method of claim 15 , wherein generating the inpainted digital image in the second resolution comprises determining, from the first nearest neighbor field, a second nearest neighbor field defining a second set of pixel offsets for inpainting in the second resolution.

17. The computer-implemented method of claim 16 , wherein determining the second nearest neighbor field for inpainting in the second resolution comprises:
selecting, from a hierarchy of scales associated with the patch match algorithm, a target scale corresponding to the first resolution; and
initializing the patch match algorithm at the target scale to avoid determining pixel offsets at resolutions lower than the first resolution.

18. The computer-implemented method of claim 14 , wherein determining the results from the patch match algorithm comprises determining pixel offsets for filling pixels of a thumbnail version of a stitched panoramic digital image.

19. The computer-implemented method of claim 14 , wherein generating the inpainted digital image comprises generating a panoramic digital image in the second resolution utilizing the results of the patch match algorithm to stitch two or more digital images together.

20. The computer-implemented method of claim 14 , wherein generating the inpainted digital image comprises initializing the patch match algorithm utilizing a nearest neighbor field determined from the digital image in the first resolution.

Assignees

• Adobe Inc

Inventors

• Amirghodsi Sohrab
• Darabi Aliakbar
• Shechtman Elya

Classifications

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06T9/00

  Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T7/0002

  Inspection of images, e.g. flaw detection · CPC title