Device, system, and method of blind deblurring and blind super-resolution utilizing internal patch recurrence

Patent metadata

Field	Value
Publication number	US-9692939-B2
Application number	US-201414289670-A
Country	US
Kind code	B2
Filing date	May 29, 2014
Priority date	May 29, 2013
Publication date	Jun 27, 2017
Grant date	Jun 27, 2017

Abstract

Devices, systems, and methods of blind deblurring and blind super-resolution utilizing internal patch recurrence. Small signal patches tend to repeat “as is” across multiple scales of a natural signal. This fractal-like behavior is utilized for signal processing tasks, including “Blind Deblurring” or “Blind Super-Resolution”, namely, removing signal blur or increasing signal resolution without a-priori knowledge of the underlying blur kernel. While the cross-scale patch recurrence is strong in signals taken under ideal conditions, the cross-scale patch recurrence significantly diminishes when the acquisition blur deviates from an ideal blur. These deviations from ideal patch recurrences are used for recovering the underlying (unknown) blur kernel. The correct blur kernel is recovered by seeking the kernel which maximizes the patch similarity across scales of a related “reference” signal. For example, this reference signal may be the low-resolution input signal, the sharp deblurred-version of a blurry input signal, or the like. Quantitative and qualitative experiments indicate that this approach yields improved or superior results, in “Blind Deblurring” and in “Blind Super-Resolution”.

First claim

What is claimed is:

1. A computer implemented method for performing a signal-processing task of blind deblurring on an input signal, the method comprising:
estimating a blur kernel associated with said input signal, by utilizing recurrence of signal patches across multiple scales inside a reference signal, wherein said reference signal is a related version of said input signal, and said blur kernel is a blur function relating said input signal and a sharper output version of said input signal having same scale; and
utilizing said estimate of said blur kernel for deblurring said input signal;
wherein said estimating comprises steps of:
generating a down-scaled signal by down-scaling said reference signal;
generating an estimate of said sharper output; and
generating said estimate of said blur kernel;

wherein patches in said estimate of said sharper output are similar to patches in said down-scaled signal;
and wherein convolving said estimate of said sharper output with said estimate of said blur kernel is similar to said input signal.

2. The method of claim 1 , wherein said reference signal is at least one of:
said input signal,
a scaled-down version of said input signal,
a deblurred version of said input signal,
a sharper version of said input signal,
a higher-quality version of said input signal,
a filtered version of said input signal,
a transformed version of said input signal.

3. The method of claim 1 , wherein said input signal is at least one of:
an image, a photograph, a video sequence, an animation sequence, a multimedia clip, a medical signal, a Magnetic Resonance Imaging (MRI) signal, a functional MRI signal, an audio signal, a single-dimensional signal, a multi-dimensional signal.

4. The method of claim 1 , wherein said signal patches are at least one of:
spatial patches, space-time patches, temporal patches, single-dimensional patches, multi-dimensional patches.

5. The method of claim 1 , wherein said reference signal is a transformed version of said input signal; wherein said transformed version of said input signal is obtained by one or more of the following transformations:
scaling-down, scaling-up, replication, interpolation, rotation, shear, a homography, an affine transformation, a parametric transformation, a non-parametric transformation, a geometric transformation, a photometric transformation, a color transformation, an intensity transformation, a transformation to a descriptor space, a Fourier Transformation, a short-time Fourier Transform, a wavelet transform, a spectral transformation, time scaling using a phase vocoder, a transformation to another representation space.

6. The method of claim 1 , wherein said blur kernel is at least one of:
an optical blur,
a camera Point Spread Function (PSF)
a temporal exposure blur,
a motion blur,
a degradation kernel,
a convolution kernel.
a transform kernel,
a uniform kernel,
a non-uniform kernel,
a sensor acquisition kernel,
a reverberation kernel.

7. The method of claim 1 , further comprising performing at least one of:
signal deblurring utilizing said blur kernel,
signal enhancement utilizing said blur kernel,
signal dereverberation utilizing said blur kernel,
sensor characterization utilizing said blur kernel,
sensor identification utilizing said blur kernel.

8. The method of claim 1 , wherein said method is implemented by a device selected from the group consisting of:
a computer, a desktop computer, a laptop computer, a server, a cloud-based computing server, a cloud-based computing system, a Software as a Service (SaaS) system, a workstation, a tablet, a smartphone, a mobile electronic device, a cellular phone, a portable electronic device, a wearable electronic device, a gaming device, a gaming console, a digital camera, a video camera, an imager, a medical imaging device, an image editing apparatus, an image processing apparatus, a photo editing apparatus, a medical imaging apparatus, a medical imaging system.

9. The method of claim 1 , wherein said generating said estimate of said blur kernel comprising iteration algorithm steps of:
(a) defining an initial estimate of said blur kernel;
(b) defining an initial estimate of said reference signal;
(c) generating a down-scaled signal by down-scaling said estimate of said reference signal;
(d) generating an updated estimate of said reference signal, wherein patches in said updated estimate of said reference signal are similar to patches in said down-scaled signal and also wherein convolving said updated estimate of said reference signal with said estimate of said blur kernel is similar to said input signal;
(e) updating said estimate of said blur kernel to be the blur kernel which best relates said updated estimate of said reference signal and said input signal;
(f) iteratively repeating steps (c) through (e) with said updated estimate of said blur kernel and said updated estimate of said reference signal.

10. The method of claim 9 , wherein said generating said updated estimate of said reference signal further comprises iteration algorithm steps of:
(d1) composing an intermediate signal by replacing patches in current said estimate of said reference signal with a weighted average of their similar patches within said down-scaled signal, while averaging overlaps;
(d2) updating said estimate of said reference signal to be a solution to a least-squares problem containing the following two quadratic terms: (i) a term which enforces that said updated estimate of said reference signal to be close to said intermediate signal, and (ii) a term which enforces that the convolution of said updated estimate of said reference signal with current said estimate of said blur-kernel be close to said input signal;
(d3) iteratively repeating steps (d1) and (d2) with said updated estimate of said reference signal.

11. The method of claim 9 , wherein said initial estimate of said blur kernel is a delta function; and wherein said initial estimate of said reference signal is said input signal.

12. The method of claim 1 , wherein said recurrence of signal patches is detected by one or more of the following similarity measures: normalized correlation, Lp-norm, mutual information, Sum of Square Differences (SSD), and mean-square-error.

13. The method of claim 1 , wherein said recurrence of signal patches is determined by comparing one or more of the following patch properties: patch colors, patch intensities, patch frequency content, patch Laplacians, patch gradients, patch derivatives, filtered versions of the patch, high-pass filtered versions of the patch, low-pass filtered versions of the patch, band-pass filtered versions of the patch, patch descriptors, a function applied to the patch, features of the patch.

14. The method of claim 1 , further comprising: detecting recurrence of signal patches by an approximate Nearest Neighbor search.

15. The method of claim 1 , wherein said recurrence of signal patches is searched locally within a constrained portion of said reference signal.

16. The method of claim 9 , wherein said estimate of said sharper output is set to be at least one of:
the final said updated estimate of said reference signal; and
the output of any non-blind deblurring method by providing it the final said estimate of said blur kernel.

17. A device for performing a signal-processing task of blind deblurring on an input signal, the device comprising: at least one processor in communication with at least one memory unit, wherein said at least one processor is configured for:
estimation of a blur kern

Assignees

• Yeda Res & Dev
• Yeda Res & Dev

Inventors

• Irani Michal
• Michaeli Tomer

Classifications

• H04N1/4092 Primary

  Edge or detail enhancement · CPC title

• G06T5/003

  Physics · mapped topic

• G06T2207/20192

  Edge enhancement; Edge preservation · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T5/73

  Deblurring; Sharpening · CPC title