Generating modified digital images utilizing nearest neighbor fields from patch matching operations of alternate digital images

Patent metadata

Field	Value
Publication number	US-11449974-B2
Application number	US-201916678132-A
Country	US
Kind code	B2
Filing date	Nov 8, 2019
Priority date	Nov 8, 2019
Publication date	Sep 20, 2022
Grant date	Sep 20, 2022

Abstract

The present disclosure relates to systems, methods, and non-transitory computer readable media for generating modified digital images by utilizing a patch match algorithm to generate nearest neighbor fields for a second digital image based on a nearest neighbor field associated with a first digital image. For example, the disclosed systems can identify a nearest neighbor field associated with a first digital image of a first resolution. Based on the nearest neighbor field of the first digital image, the disclosed systems can utilize a patch match algorithm to generate a nearest neighbor field for a second digital image of a second resolution larger than the first resolution. The disclosed systems can further generate a modified digital image by filling a target region of the second digital image utilizing the generated nearest neighbor field.

First claim

What is claimed is:

1. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a computer device to:
identify a first image nearest neighbor field comprising a first plurality of pixel offsets associated with a first digital image corresponding to a first resolution;
utilize the first image nearest neighbor field associated with the first digital image to determine a second image nearest neighbor field comprising a second plurality of pixel offsets for a second digital image corresponding to a second resolution;
determine a digital image patch for filling a region of the second digital image based on the second image nearest neighbor field for the second digital image; and
generate a modified digital image utilizing the digital image patch.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer device to determine a target scale within a hierarchy of scales based on the first resolution of the first digital image; and
wherein the second resolution is larger than the first resolution.

3. The non-transitory computer readable medium of claim 2 , further comprising instructions that, when executed by the at least one processor, cause the computer device to:
determine that the first resolution of the first digital image does not match any scale within the hierarchy of scales; and
upsample, based on determining that the first resolution does not match any scale within the hierarchy of scales, the first image nearest neighbor field based on the first resolution of the first digital image and the target scale of the hierarchy of scales.

4. The non-transitory computer readable medium of claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computer device to utilize the upsampled first image nearest neighbor field associated with the first digital image as the second image nearest neighbor field at the target scale.

5. The non-transitory computer readable medium of claim 3 , further comprising instructions that, when executed by the at least one processor, cause the computer device to refrain from determining nearest neighbor fields for the second digital image at scales smaller than the target scale within the hierarchy of scales.

6. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer device to determine the second image nearest neighbor field by determining pixel offsets that indicate pixels to match the region of the second digital image.

7. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer device to:
generate a compressed nearest neighbor field for the second digital image by merging regions of the second image nearest neighbor field together based on similarities between the regions; and
store the compressed nearest neighbor field for the second digital image.

8. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computer device to generate the modified digital image by filling the region of the second digital image utilizing pixels indicated by the second image nearest neighbor field.

9. A system comprising:
one or more memory devices comprising a first digital image corresponding to a first resolution, a first image nearest neighbor field corresponding to the first digital image, and a second digital image corresponding to a second resolution; and
one or more computing devices that cause the system to utilize the first image nearest neighbor field to generate a digital image patch for the second digital image by:
determining, based on the first resolution, a target scale within a hierarchy of scales corresponding to a patch match algorithm;
utilizing the first image nearest neighbor field to determine a second image nearest neighbor field for the second digital image at the target scale;
determining, based on the second image nearest neighbor field at the target scale, a subsequent-scale nearest neighbor field for the second digital image at an additional scale within the hierarchy of scales; and
generating the digital image patch based on the subsequent-scale nearest neighbor field.

10. The system of claim 9 , wherein the one or more computing devices further cause the system to initialize the patch match algorithm at the target scale to determine the second image nearest neighbor field based on the first image nearest neighbor field.

11. The system of claim 9 , wherein the one or more computing devices further cause the system to determine colors for the digital image patch by performing a voting operation based on the subsequent-scale nearest neighbor field for the second digital image.

12. The system of claim 9 , wherein determining the target scale comprises comparing the first resolution of the first digital image with designated scales within the hierarchy of scales to determine a closest larger designated scale relative to the first resolution as the target scale.

13. The system of claim 9 , wherein generating the digital image patch comprises identifying pixels to fill a target region associated with the second digital image.

14. The system of claim 9 , wherein the first digital image comprises a thumbnail digital image and the second digital image comprises a high-resolution digital image.

15. The system of claim 9 , wherein the one or more computing devices further cause the system to:
determine that the first resolution does not match any scale within the hierarchy of scales; and
based on determining that the first resolution does not match any scale within the hierarchy of scales, upsample the first image nearest neighbor field to match the target scale.

16. The system of claim 9 , wherein the second resolution of the second digital image is larger than the first resolution of the first digital image.

17. A computer-implemented method comprising:
identifying a first image nearest neighbor field comprising a first plurality of pixel offsets associated with a first digital image corresponding to a first resolution;
utilizing the first image nearest neighbor field associated with the first digital image to determine a second image nearest neighbor field comprising a second plurality of pixel offsets for a second digital image corresponding to a second resolution;
determining a digital image patch for filling a region of the second digital image based on the second image nearest neighbor field for the second digital image; and
generating a modified digital image utilizing the digital image patch.

18. The computer-implemented method of claim 17 , wherein the second resolution is different from the first resolution.

19. The computer-implemented method of claim 17 , wherein identifying the first image nearest neighbor field associated with the first digital image comprises identifying a plurality of offsets that indicate pixels to match a target region of the first digital image.

20. The computer-implemented method of claim 17 , wherein generating the modified digital image comprises filling a target region associated with the second digital image utilizing the digital image patch.

Assignees

• Adobe Inc

Inventors

• Amirghodsi Sohrab
• Darabi Aliakbar
• Shechtman Elya

Classifications

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T7/0002 Primary

  Inspection of images, e.g. flaw detection · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2200/32

  involving image mosaicing · CPC title