Method for generating high-resolution images using regression patterns

Patent metadata

Field	Value
Publication number	US-9734558-B2
Application number	US-201414220402-A
Country	US
Kind code	B2
Filing date	Mar 20, 2014
Priority date	Mar 20, 2014
Publication date	Aug 15, 2017
Grant date	Aug 15, 2017

Abstract

A method generates a high-resolution (HR) image from a low-resolution (LR) image using regression functions. During a training stage, training HR images are downsampled to LR images. A signature is determined for each LR-HR patch pair based on a local ternary pattern (LTP). The signature is a low dimensional descriptor used as an abstraction of the patch pair features. Then, patch pairs with the same signature are clustered, and a regression function which maps the LR patches to the HR patches is determined. In some cases patch pairs of similar signatures can be combined for learning and a single regression function determined, thus decreasing the number of required regression functions. During actual upscaling, LR patches of an input image are similarly processed to obtain the signatures and from the regression functions. The LR patches can then be upscaled using the training regression functions.

First claim

We claim:

1. A method for processing input low-resolution (LR) images to output high-resolution (HR) images, wherein the LR images and the HR images include corresponding LR-HR patch pairs, such that each patch includes pixel values, the method comprising:
performing, by a processor, an offline training stage by:
acquiring stored historical LR-HR patch pairs from a memory in communication with the processor, wherein each LR patch is a downsampled version of a corresponding HR patch of the same LR-HR patch pair;
calculating a local n-ary pattern (LnP) for each LR-HR patch pair for all the LR-HR patch pairs, and grouping all LR-HR patch pairs with the same LnP value together; and
obtaining for each LnP value a trained regression function appropriate for upscaling patches of that LnP value, and storing the trained regression function for that LnP value;

performing, by the processor, a real-time upscaling stage of an input LR image by:
acquiring each stored LR input image and processing each patch of the LR input image by:
calculating a LnP value for each input LR patch, then using the LnP value to select a trained regression function for that patch, and then applying the selected trained regression function to that patch to obtain a corresponding HR patch;
repeating the process for the remaining patches of the LR input image; and
obtaining the HR output image from overlapping HR patches by averaging HR patch pixel values where HR patches overlap.

2. The method of claim 1 , wherein the LnP is a local binary pattern.

3. The method of claim 1 , wherein the processing operates on sequences of images.

4. The method of claim 1 , wherein the patches overlap.

5. The method of claim 1 , wherein the enlarging uses bicubic upsampling.

6. The method of claim 1 , wherein the regression function is a linear regression function.

7. The method of claim 1 , wherein the HR image patches are predetermined.

8. The method of claim 1 , wherein selecting the regression function based on the determined LnP for each ELR image patch is based, in part, on overlapping patches, such that an amount of overlap between adjacent patches is one of application specific or adaptive.

9. The method of claim 1 , wherein the LnP is a local ternary pattern.

10. The method of claim 9 , wherein the LR image is preprocessed using bicubic upsampling to produce an enlarged LR (ELR), such that the ELR is used to replace of the LR image.

11. The method of claim 10 , further comprising:
partitioning each ELR image patch into a 3×3 cells of 9 pixels; and
comparing an intensity of a center cell to intensities of eight neighboring cells, beginning with an upper left neighboring cell, and moving clockwise around the center cell, and if the intensity is equal to or greater than the intensity of the center cell plus a threshold, then assign +1, if the intensity is equal to or smaller than the intensity of the center cell minus the threshold, then assign −1, otherwise assign a 0 according to

{

1
,

if
⁢

⁢
p

≥

c
+
k

0
,

if
⁢

⁢
p

>

c
-

k
⁢

⁢
and
⁢

⁢
p

<

c
+
k

-
1

if
⁢

⁢
p

<

c
-
k

,

where p is the intensity of the cell, c is the intensity of the center cell and k is the threshold;
clustering the patch pairs with the same signature into a same group; and
learning the regression function for each group.

12. The method of claim 11 , further comprising:
combining the patch pairs of selected groups based on a similarity criterion; and
learning a single function from the combined patch pairs.

13. The method of claim 12 , wherein the similarity criterion is based on rotated, symmetric, or rotated and symmetric versions of the LnPs.

14. The method of claim 11 , wherein selected groups are combined to train a generic regression function, wherein the selected groups occur less than some predetermined threshold.

15. The method of claim 11 , further comprising:
combining the patch pairs with symmetric LnPs in the same group;
combining the patch pairs with rotated LnPs in the same group;
combining the patch pairs with rotated and symmetric LnPs in the same group;
choosing a reference LnP for each group;
determining a lookup table from the LnPs to the reference LnP in the same group;
transforming patch pairs in the same group by spatial rotation and mirroring operations to all pairs have the reference LnP; and
learning the regression function for each group.

16. A system for processing input low-resolution (LR) images to output high-resolution (HR) images, wherein the LR images and the HR images include corresponding LR-HR patch pairs, such that each patch includes pixel values, the system comprising:
a processor; and
a memory coupled to the processor and storing program instructions executable by the processor to implement, the processor is config

Assignees

• Mitsubishi Electric Res Laboratories Inc

Inventors

• Porikli Fatih
• Shrivastava Ashish
• Thornton Jay

Classifications

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/30168

  Image quality inspection · CPC title