Image processor for and method of upscaling and denoising using contextual video information

Patent metadata

Field	Value
Publication number	US-9235874-B2
Application number	US-201314067409-A
Country	US
Kind code	B2
Filing date	Oct 30, 2013
Priority date	Apr 8, 2013
Publication date	Jan 12, 2016
Grant date	Jan 12, 2016

Abstract

High quality upscaling and denoising are required in mobile imaging devices that do not contain high quality lenses. Such is also required in order to scale up standard-definition video content for display in high-definition television screens.
The disclosed method uses contextual information obtained during upscaling and/or denoising of frames. Relevant correspondences between patches within a frame and between frames, are detected, managed and exploited. The correspondence information is simultaneously used and updated while video frames are being processed. Two approaches may be used: 1. keeping, searching for and updating a database of useful patches, by adding frequently visible similar patches, aggregating high-frequency, low-noise information associated with the similar patches, and removing less-observed patches; 2. Using the high-resolution and noise-reduced information that was collected from earlier video frames, and is expressed by the output of latest processed frame, for upscaling and/or noise-reducing the next processed frame.

First claim

What is claimed is:

1. An image processing method of an image processor comprising:
storing a plurality of patches being frame fragments, the size of each frame fragment being smaller than that of a frame;
upscaling an input frame based on a patch dictionary unit;
updating the patch dictionary unit according to an upscale result,
wherein while upscaling of the input frame, a high frequency component of the upscaled input frame is reinforced based on the patch dictionary unit, and
wherein the updating the patch dictionary according to an upscale result comprises:
adding frequently-visible similar patches in the upscaled input frame to the patch dictionary unit;
aggregating high-frequency and low-noise information associated with a plurality of patches in the patch dictionary unit; and
removing less-observed patches in the upscaled input frame from the patch dictionary unit.

2. The image processing method of claim 1 , wherein the patch dictionary unit is updated based on at least one patch used for performing the upscaling.

3. The image processing method of claim 1 ,
wherein the upscaling an input frame comprises:
selecting a magnified patch of the upscaled input frame;
detecting a descriptor indicating a low frequency component of the magnified patch from the magnified patch;
searching for a first patch in a relevant area of the input frame based on the descriptor;
searching for a second patch in the patch dictionary unit based on the descriptor, and
modifying the magnified patch using one, closer to the magnified patch, from among the first and second patches.

4. The image processing method of claim 3 , wherein the updating the patch dictionary unit according to an upscale result comprises:
calculating the difference between the first patch and the second patch;
replacing the second patch stored in the patch dictionary unit with the first patch if the calculated difference is less than a threshold value;
determining whether the difference between the first patch and the magnified patch is less than the difference between the second patch and the magnified patch, if the calculated difference is not less than a threshold value;
adding the first patch to the patch dictionary unit if the first patch is closer to the magnified patch than the second patch; and
ignoring the first patch if the difference between the first patch and the magnified patch is not less than the difference between the second patch and the magnified patch.

5. The image processing method of claim 4 , wherein the updating the patch dictionary unit according to an upscale result further comprises:
increasing a score of a patch, used at the upscaling, from among patches stored in the patch dictionary unit;
assigning a predetermined score to a new patch if the new patch is added to the patch dictionary unit; and
decreasing the scores of patches stored in the patch dictionary unit if the predetermined number of search operations is executed at the patch dictionary unit.

6. The image processing method of claim 5 , wherein the updating the patch dictionary unit according to an upscale result further comprises: after the scores are decreased, deleting a patch having a score lower than a threshold value from among patches stored in the patch dictionary unit.

7. The image processing method of claim 5 , wherein the updating the patch dictionary unit according to an upscale result further comprises: if the size of the patch dictionary unit reaches a predetermined limit value upon adding of a new patch to the patch dictionary unit, deleting a patch having a score lower than a threshold value from among patches stored in the patch dictionary unit.

8. The image processing method of claim 4 , wherein the updating the patch dictionary unit according to an upscale result further comprises: if the number of the similar patches similar to the new patch, from among patches stored in the patch dictionary unit is larger than a threshold value upon adding of the new patch to the patch dictionary unit, then combining the new patch and the similar patches.

9. The image processing method of claim 1 , wherein if the patch dictionary unit is initialized. remaining patches other than baseline patches are deleted from the patch dictionary unit.

10. The image processing method of claim 1 ,
wherein the upscaling an input frame comprises:
selecting a magnified patch of the upscaled input frame;
searching in a relevant area of the upscaled input frame for at least one first patch similar to the magnified patch;
searching in the patch dictionary unit for a second patch similar to the magnified patch;
executing refinement using the at least one first patch and the second patch; and
replacing the magnified patch with the refined patch.

11. The image processing method of claim 10 , wherein the updating the patch dictionary unit according to an upscale result comprises:
if the second patch is searched for in the patch dictionary unit then increasing a score of the second patch;
adding the at least one first patch to the patch dictionary unit with a first initial score if the second patch is not searched for in the patch dictionary unit and the at least one first patch is searched for; and
adding the magnified patch to the patch dictionary unit with a second initial score smaller than the first initial score if the second patch is not searched for in the patch dictionary unit and the at least one first patch is not searched for.

12. The image processing method of claim 1 , further comprising:
downscaling a previous frame upscaled;
generating a motion field by performing motion detection on the downscaled frame and the input frame; and
generating a upscaled partial frame by copying patches of the previous frame upscaled, based on the motion field,
wherein the upscaling is executed with respect to the upscaled partial frame.

13. The image processing method of claim 1 , wherein the upscaling an input frame based on a patch dictionary unit comprises: upscaling or noise-reducing the input frame using the high-frequency and low-noise information that have been collected from a plurality of previous input frames.

14. An image processor comprising:
a patch dictionary unit for storing a plurality of patches being frame fragments, the size of each frame fragment being smaller than the size of a frame;
an upscale unit configured to receive an input frame, to upscale the input frame, and to search or patches of the upscaled input frame and of the patch dictionary unit;
a dictionary management unit configured to update the patch dictionary unit, and
a patch refinement unit configured to execute refinement of a patch of the upscaled input frame and a patch of the patch dictionary unit searched for by the upscale unit,
wherein the upscale unit executes an upscale on the input frame using the refined patches and the dictionary management unit updates the patch dictionary based on the refined patches.

15. The image processor of claim 14 , further comprising:
a downscale unit configured to downscale an output frame of the upscale unit;
a motion estimation unit configured to perform motion estimation using a frame downscaled by the downscale unit and an initial input frame; and
a motion compensation unit configured to generate a upscaled partial frame by copying patches of the output frame based on a motion field estimated by the motion estimation unit, the upscaled partial frame being sent to the upscale unit as the input frame.

16. An image processor comprising:
a motion estimation unit configured to perform motion esti

Assignees

• Samsung Electronics Co Ltd

Inventors

• Voronov German
• Litvak Shai
• Yanir Tomer
• Brailovsky Leonid

Classifications

• G06T2207/20182

  Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title