Super-resolution with reference images

Patent metadata

Field	Value
Publication number	US-10885608-B2
Application number	US-201816001656-A
Country	US
Kind code	B2
Filing date	Jun 6, 2018
Priority date	Jun 6, 2018
Publication date	Jan 5, 2021
Grant date	Jan 5, 2021

Abstract

In implementations of super-resolution with reference images, a super-resolution image is generated based on reference images. Reference images are not constrained to have same or similar content as a low-resolution image being super-resolved. Texture features indicating high-frequency content are extracted into texture feature maps, and patches of texture feature maps of reference images are determined based on texture feature similarity. A content feature map indicating low-frequency content of an image is adaptively fused with a swapped texture feature map including patches of reference images with a neural network based on similarity of texture features. A user interfaces allows a user to select regions of multiple reference images to use for super-resolution. Hence, a super-resolution image can be generated with rich texture details incorporated from multiple reference images, even in the absence of reference images having similar content to an image being upscaled.

First claim

What is claimed is:

1. In a digital medium environment to generate super-resolution images, a method implemented by a computing device, the method comprising:
extracting, by the computing device, content features from a low-resolution image into a content feature map;
extracting, by the computing device, first texture features from a scaled version of the low-resolution image into a first texture feature map and second texture features from at least one reference image into a second texture feature map, the first texture feature map and the second texture feature map including first patches and second patches, respectively;
generating, by the computing device, a swapped texture map by replacing each first patch in the first texture feature map with a respective matching second patch from the second texture feature map, the respective matching second patch for each first patch determined from inner product values generated between pairs of each first patch and the second patches, and the respective matching second patch selected from one of the pairs having a largest inner product value; and
combining, by the computing device, the content feature map and the swapped texture map according to a respective similarity score between said each first patch and the respective matching second patch to form a super-resolution image.

2. The method as described in claim 1 , wherein the content features represent lower frequency content of the low-resolution image than the first texture features.

3. The method as described in claim 1 , wherein the scaled version of the low-resolution image is formed by upscaling the low-resolution image to a resolution of the at least one reference image.

4. The method as described in claim 1 , wherein the respective similarity score between each first patch and the respective matching second patch is based on the largest inner product value.

5. The method as described in claim 1 , wherein the at least one reference image is obtained without constraining the at least one reference image to include same or similar content as the low-resolution image.

6. The method as described in claim 1 , wherein the combining includes:
comparing the respective similarity score to a similarity threshold;
accepting the respective matching second patch for inclusion in the combining to form the super-resolution image when the comparing indicates the respective similarity score is greater than the similarity threshold; and
rejecting the respective matching second patch for inclusion in the combining to form the super-resolution image when the comparing indicates the respective similarity score is not greater than the similarity threshold.

7. The method as described in claim 1 , wherein the combining includes:
multiplying the swapped texture map by a weight map indicating the respective similarity score between said each first patch and the respective matching second patch;
fusing results of the multiplying with the content feature map; and
upscaling results of the fusing to form the super-resolution image.

8. The method as described in claim 1 , wherein the at least one reference image includes multiple reference images.

9. The method as described in claim 1 , wherein the combining is performed with a neural network of the computing device, the neural network trained with low-resolution images obtained by downsampling randomly-cropped patches of a high-resolution image set and reference images obtained from other patches of the high-resolution image set than the randomly-cropped patches.

10. In a digital medium environment to generate super-resolution images, a system comprising:
a content extractor module implemented at least partially in hardware of a computing device to extract content features from a first image into a content feature map;
a texture extractor module implemented at least partially in hardware of the computing device to extract first texture features from the first image into a first texture feature map indicating high-frequency content of the first image and second texture features from a region of a reference image into a second texture feature map indicating high-frequency content of the region of the reference image;
a patch matching module implemented at least partially in hardware of the computing device to determine a respective patch of the second texture feature map for each patch of the first texture feature map;
a weight map module implemented at least partially in hardware of the computing device to generate a weight map indicating a respective similarity score between the respective patch of the second texture feature map and said each patch of the first texture feature map;
a texture swapping module implemented at least partially in hardware of the computing device to generate a swapped texture map by replacing said each patch in the first texture feature map with the respective patch from the second texture feature map; and
a texture transfer module implemented at least partially in hardware of the computing device to combine the content feature map and the swapped texture map according to the weight map to form a super-resolution image, the content feature map and the swapped texture map combined according to the weight map with a neural network trained according to a loss function regularized by texture consistency between the super-resolution image and the reference image.

11. The system as described in claim 10 , wherein the region of the reference image is determined from a user selection that indicates to crop the reference image to designate the region of the reference image.

12. The system as described in claim 10 , wherein the texture swapping module is further configured to average areas of the swapped texture map overlapped by multiple respective patches from the second texture feature map that are swapped into the swapped texture map.

13. The system as described in claim 10 , wherein the patch matching module determines the respective patch of the second texture feature map for each patch of the first texture feature map by convolving the second texture feature map with patches of the second texture feature map.

14. The system as described in claim 10 , wherein the first image is an upscaled version of a low-resolution image.

15. In a digital medium environment to generate super-resolution images, a method implemented by a computing device, the method comprising:
a step for exposing a user interface including a first image and a reference image;
a step for receiving a user selection via the user interface of a region of the reference image;
a step for extracting content features from the first image into a content feature map;
a step for extracting first texture features from the first image into a first texture feature map indicating high-frequency content of the first image;
a step for extracting second texture features from the region of the reference image into a second texture feature map indicating high-frequency content of the region of the reference image;
a step for determining, for each first patch of the first texture feature map, a respective matching second patch of the second texture feature map;
a step for generating a weight map indicating a respective similarity score between said each first patch and the respective matching second patch, the respective similarity score determined from an inner product between each first patch normalized by a norm of each first patch and the respective matching second patch normalized by a norm of the respective matching second patch;
a step for generating a swapped texture map by replacing said each fi

Assignees

• Adobe Inc

Inventors

• Zhang Zhifei
• Lin Zhe
• Wang Zhaowen

Classifications

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T3/4061 Primary

  by injecting details from different spectral ranges · CPC title

• G06V10/54

  relating to texture · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title