Content aware sampling during patch synthesis

Patent metadata

Field	Value
Publication number	US-10134108-B2
Application number	US-201615286245-A
Country	US
Kind code	B2
Filing date	Oct 5, 2016
Priority date	Oct 5, 2016
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

Embodiments of the present invention provide systems, methods, and computer storage media directed at image synthesis utilizing sampling of patch correspondence information between iterations at different scales. A patch synthesis technique can be performed to synthesize a target region at a first image scale based on portions of a source region that are identified by the patch synthesis technique. The image can then be sampled to generate an image at a second image scale. The sampling can include generating patch correspondence information for the image at the second image scale. Invalid patch assignments in the patch correspondence information at the second image scale can then be identified, and valid patches can be assigned to the pixels having invalid patch assignments. Other embodiments may be described and/or claimed.

First claim

What is claimed is:

1. One or more non-transitory computer-readable storage media having instructions stored thereon, which, when executed by one or more processors of a computing device, provide the computing device with a graphics editor to:
perform a patch synthesis process on a target region of an image at a first image scale to determine first patch correspondence information for pixels in the target region at the first image scale, the first patch correspondence information comprising a first map of patches from a source region assigned to pixels in the target region at the first image scale;
generate the image at a second image scale based on sampling of the image at the first image scale;
generate second patch correspondence information for pixels in the target region at the second image scale, the second patch correspondence information comprising a second map of patches from the source region assigned to the pixels in the target region at the second image scale, wherein one or more pixels in the target region have an invalid patch assignment based on at least a scaling operation from the first image scale to the second image scale;
perform an incremental search on at least one pixel of the one more pixels having an invalid patch assignment to identify a pixel having a valid patch assignment; and
assign to the at least one pixel a patch corresponding to the valid patch assignment for the identified pixel.

2. The one or more computer-readable storage media of claim 1 , wherein the instructions provide the computing device with a graphics editor to further generate a validity mask that indicates pixels having an invalid patch assignment and pixels having a valid patch assignment.

3. The one or more computer-readable storage media of claim 2 , wherein the validity mask further indicates pixels located outside of the target region, the indication for pixels located outside of the target region being different from the indication for pixels having an invalid patch assignment and the indication for pixels having a valid patch assignment.

4. The one or more computer-readable storage media of claim 1 , wherein the instructions provide the computing device with a graphics editor to further perform a second patch synthesis process on the target region of the image at the second image scale to determine third patch correspondence information for pixels in the target region at the second image scale, the third patch correspondence information comprising a third map of patches from the source region assigned to pixels in the target region at the second image scale.

5. The one or more computer-readable storage media of claim 4 , wherein the instructions provide the computing device with a graphics editor to further generate an active mask that indicates one or more portions of the target region as inactive, the one or more portions being composed of groupings of pixels that have converged, wherein at least a portion of the second patch synthesis process ignores the one or more portions indicated as inactive by the active mask.

6. The one or more computer-readable storage media of claim 1 , wherein the image at the second scale comprises a higher pixel density than the image at the first scale.

7. The one or more computer-readable storage media of claim 1 , wherein the image comprises the source region.

8. The one or more computer-readable storage media of claim 1 , wherein the image comprises the target region, the source region, and at least one additional pixel.

9. The one or more computer-readable storage media of claim 1 , wherein to perform a patch synthesis process includes:
performance of an approximate nearest-neighbor search process to identify the portions of the source region that are to be utilized in synthesizing the target region; and
performance of a voting process, based on the identified portions of the source region, to generate colors for individual pixels within the target region.

10. The one or more computer-readable storage media of claim 1 , wherein the incremental search comprises a concentric shell search to identify a pixel in a concentric shell having a valid patch assignment.

11. The one or more computer-readable storage media of claim 1 , wherein the first patch correspondence information comprises a nearest-neighbor field.

12. A computer-implemented method for synthesizing digital images comprising:
performing a patch synthesis process on a target region of an image at a first image scale to determine first patch correspondence information for pixels in the target region at the first image scale, the first patch correspondence information comprising a first map of patches from a source region assigned to pixels in the target region at the first image scale;
generating the image at a second image scale based on sampling of the image at the first image scale;
generating second patch correspondence information for pixels in the target region at the second image scale, the second patch correspondence information comprising a second map of patches from the source region assigned to pixels in the target region at the second image scale, wherein one or more pixels in the target region have an invalid patch assignment based on at least a scaling operation from the first image scale to the second image scale;
generating a validity mask that indicates pixels having an invalid patch assignment and pixels having a valid patch assignment;
performing, for at least one pixel of the one more pixels having an invalid patch assignment, a distance evaluation between the at least one pixel and a plurality of pixels having a valid patch assignment in the validity mask; and
assigning, to the at least one pixel, a patch corresponding to a nearest neighbor pixel based on the performed distance evaluation.

13. The computer-implemented method of claim 12 , wherein the validity mask further indicates pixels located outside of the target region, the indication for pixels located outside of the target region being different from the indication for pixels having an invalid patch assignment and the indication for pixels having a valid patch assignment.

14. The computer-implemented method of claim 12 , wherein performing a distance evaluation comprises computing a distance transform for the at least one pixel, the distance transform being computed prior to at least one of performing the patch synthesis process on the target region of the image at the first image scale and generating the image at the second image scale based on sampling of the image at the first scale.

15. The computer-implemented method of claim 12 , further comprising performing a second patch synthesis process on the target region of the image at the second image scale to determine third patch correspondence information for pixels in the target region at the second image scale, the third patch correspondence information comprising a third map of patches from the source region assigned to pixels in the target region at the second image scale.

16. A computing system for synthesizing digital images comprising:
one or more processors; and
one or more computer-readable storage media, coupled with the one or more processors, having instructions stored thereon, which, when executed by the one or more processors, provide the computing system with a graphics editor to:
perform a patch synthesis process on a target region of an image at a first image scale to determine first patch correspondence information for pixels in the target region at the first image scale, the first patch correspondence information comprising a first map of patches fro

Assignees

• Adobe Systems Inc

Inventors

• Amirghodsi Sohrab
• Darabi Aliakbar
• Shechtman Elya

Classifications

• G06T3/40 Primary

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T7/0081

  Physics · mapped topic

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T7/73

  using feature-based methods · CPC title