Visual-experience-optimized super-resolution frame generator

Patent metadata

Field	Value
Publication number	US-9547887-B2
Application number	US-201414163076-A
Country	US
Kind code	B2
Filing date	Jan 24, 2014
Priority date	Sep 26, 2013
Publication date	Jan 17, 2017
Grant date	Jan 17, 2017

Abstract

An image processor generates a Super-Resolution (SR) frame by upscaling. A Human Visual Preference Model (HVPM) helps detect random texture regions, where visual artifacts and errors are tolerated to allow for more image details, and immaculate regions having flat areas, corners, or regular structures, where details may be sacrificed to prevent annoying visual artifacts that seem to stand out more. A regularity or isotropic measurement is generated for each input pixel. More regular and less anisotropic regions are mapped as immaculate regions. Higher weights for blurring, smoothing, or blending from a single frame source are assigned for immaculate regions to reduce the likelihood of generated artifacts. In the random texture regions, multiple frames are used as sources for blending, and sharpening is increased to enhance details, but more artifacts are likely. These artifacts are more easily tolerated by humans in the random texture regions than in the regular-structure immaculate regions.

First claim

We claim:

1. A visual-experience optimized image processor comprising:
an input receiving an input frame of pixels in a sequence of input frames having a low resolution;
an output for outputting an output frame of pixels in a sequence of output frames having a high resolution, wherein the output frame comprises at least two times a number of pixels in the input frame;
an upscaler for generating additional pixels in the output frame from pixels in the input frame using image processing routines that also generate visual artifacts that are not present in the input frame;
a region classifier that identifies structured regions and random-texture regions within a frame of pixels, the region classifier indicating a random-texture region when a measurement of anisotropy among pixels in a region is higher than a threshold, the region classifier indicating a structured region when the measurement of anisotropy among pixels in the region is lower than the threshold;
wherein the random-texture regions have a degree of anisotropy that is lower than in the structured regions; and
an optimizer that selects an image processing routine that generates fewer visual artifacts and diminishes visual details when the upscaler processes the structured regions, the optimizer selecting an image processing routine that generates more visual artifacts and enhances visual details when the upscaler processes the random-texture regions,

whereby visual artifacts and visual details are diminished for the structured regions and are enhanced for the random-texture regions in the output frame.

2. The visual-experience optimized image processor of claim 1 further comprising:
a Human-Visual-Preference Model (HVPM) map having a plurality of preference values for pixels in a frame;
wherein a preference value includes the measurement of anisotropy generated by the region classifier;
wherein the optimizer reads the HVPM map to select the image processing routine by selecting parameters that control the image processing routine or by selecting from among a plurality of image processing routines;
wherein the HVPM map controls image processing to reduce visual artifacts for the structured regions, and to enhance details and visual artifacts in the random-texture regions.

3. The visual-experience optimized image processor of claim 2 wherein the region classifier further comprises:
a gradient generator for generating a gradient for a region, the gradient indicating a measurement of flatness in the region;
a structure tensor generator for generating a tensor from the gradient, the structure tensor indicating a measure of structures within the region;
a corner detector to detect corners;
an eigenvalue generator that solves the structure tensor to generate a first eigenvalue and a second eigenvalue for the region;
a measurement generator that generates the measurement of anisotropy for the region as a degree of relative discrepancy between the first eigenvalue and the second eigenvalue; and
a human preference value generator that generates the preference values by combining the measurement of anisotropy with a Just-Noticeable Difference (JND) model.

4. The visual-experience optimized image processor of claim 2 wherein the upscaler further comprises:
a Single-Frame (SF) image processor that has a single input frame as an input to generate SF pixel values;
a Multi-Frame (MF) image processor that has multiple input frames as inputs to generate MF pixel values;
wherein the MF image processor generates more visual artifacts in a target frame than does the SF image processor when generating the target frame when using a same set of parameters;
a weight generator for generating a SF weight and a MF weight, wherein the measurement of anisotropy for the region contributes to the SF weight and to the MF weight; and
a pixel blender, receiving SF pixel values from the SF image processor and receiving MF pixels from the MF image processor, the pixel blender combining a SF pixel value scaled by the SF weight, and a MF pixel value scaled by the MF weight, to generate a pixel for the output frame;
wherein the SF pixel weight is larger than the MF pixel weight for pixels in the structured regions;
wherein the SF pixel weight is smaller than the MF pixel weight for pixels in the random-texture regions,
whereby SF and MF pixel weights are adjusted by the degree of anisotropy of the region.

5. The visual-experience optimized image processor of claim 2 further comprising:
a deblur filter that deblurs pixels in a frame in response to a deblur control input receiving a deblur control value, the deblur control input controlling a degree of deblur performed on the pixels by the deblur filter;
wherein the preference values from the HVPM map are at least a portion of the deblur control value applied to the deblur control input;
wherein pixels in the structured regions receive a lower degree of deblur than pixels in the random-texture regions when other inputs to the deblur control value are constant,
whereby structured regions are processed less sharply than random-texture regions.

6. The visual-experience optimized image processor of claim 5 further comprising:
a depth estimator that generates depth values for regions in the input frame, the depth values indicating an estimated depth from a viewer for pixels in the region;
wherein the depth values are a portion of the deblur control value applied to the deblur control input;
wherein pixels in regions having greater depth values receive a lower degree of deblur than pixels in regions having lower depth values when other inputs to the deblur control value are constant,
whereby low-depth regions are deblured more than high-depth regions.

7. The visual-experience optimized image processor of claim 2 further comprising an Iterative Back-Projection (IBP) corrector that comprises:
a smooth filter that smoothes pixels in the output frame in response to a smooth control input receiving a smooth control value to generate a smoothed intermediate frame, the smooth control input controlling a degree of smoothing performed on the pixels by the smooth filter;
wherein the preference values from the HVPM map are at least a portion of the smooth control value applied to the smooth control input;
wherein pixels in the structured regions receive a smaller degree of smoothing than pixels in the random-texture regions when other inputs to the smooth control value are constant;
a downsampler that converts the smoothed intermediate frame having the high resolution to a second intermediate frame having the low resolution;
a weighted pixel subtractor that generates residual pixels as a weighted differences between pixels in the second intermediate frame and corresponding pixels in the input frame;
an upsampler that converts the residual pixels to a high-resolution residual frame of upsampled residual pixels; and
a pixel adder that adds the upsampled residual pixels to pixels in the output frame to adjust pixels in the output frame,

whereby pixels in the output frame are adjusted by the IBP corrector that has smoothing controlled by the preference values from the HVPM map so that output pixels receive an overall effect of sharpening, and wherein output pixels in the structured regions receive a smaller degree of sharpening than output pixels in the random-texture regions when other inputs to the smooth control value are constant.

8. A method for generating Super Resolution (SR) images from Low Resolution (LR) images comprising:
building a Human Visual Preference Model (HVPM) map using a LR image, the HVPM map having a plurality of human preference values, each human preference value indicating when a pixel is in an immaculate region an

Assignees

• Hong Kong Applied Science & Tech Res Inst Company Ltd
• Hong Kong Applied Science & Tech Research Inst Co Ltd

Inventors

• Liang Luhong
• Luo Peng
• Chiu King Hung
• Cheung Wai Keung

Classifications

• G06T3/4076 Primary

  using the original low-resolution images to iteratively correct the high-resolution images · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06K9/6267

  Physics · mapped topic