Algorithm and device for image processing

Patent metadata

Field	Value
Publication number	US-9779491-B2
Application number	US-201514826961-A
Country	US
Kind code	B2
Filing date	Aug 14, 2015
Priority date	Aug 15, 2014
Publication date	Oct 3, 2017
Grant date	Oct 3, 2017

Abstract

Deblurring a blurry image ( 14 ) includes the steps of (i) computing a spatial mask ( 256 ); (ii) computing a modified blurry image ( 264 ) using the blurry image ( 14 ) and the spatial mask ( 256 ); and (iii) computing a latent sharp image ( 16 ) using the modified blurry image ( 264 ) and a point spread function ( 260 ). Additionally, the image ( 714 ) to can be analyzed to identify areas of the image ( 714 ) that are suitable for point spread function estimation. Moreover, a region point spread function ( 1630 ) can be analyzed to classify the point spread function(s) as representing (i) motion blur, (ii) defocus blur, or (iii) mixed motion blur and defocus blur. A point spread function ( 2670 ) can also be estimated.

First claim

What is claimed is:

1. A method for estimating a latent sharp image for at least a portion of a blurry image having a point spread function, the method comprising:
generating a spatial mask for the blurry image with a control system that includes a processor, the spatial mask being an array of individual weights for individual pixels;
computing a modified blurry image with the control system using the blurry image and the spatial mask; and
estimating the latent sharp image with the control system using the modified blurry image and the point spread function; and
wherein the steps of (i) generating a spatial mask, (ii) computing a modified blurry image, and (iii) estimating the latent sharp image, are alternately repeated and updated for a plurality of iterations with the control system.

2. The method of claim 1 further comprising re-blurring the latent sharp image using the point spread function to generate a re-blurred latent sharp image; and wherein the generating a spatial mask includes the spatial mask being based on a difference between the blurry image and the re-blurred latent sharp image.

3. The method of claim 1 wherein estimating the latent sharp image includes using a Wiener filter.

4. The method of claim 1 wherein estimating the latent sharp image includes using a regularized least squares cost function.

5. The method of claim 1 wherein estimating the latent sharp image includes using a generalized regularized least squares cost function.

6. A method for estimating a spatial mask that is used for deblurring a blurry image having a point spread function and a latest version of a latent sharp image, the method comprising:
generating a reblurred image with a control system that reblurs the latest version of the latent sharp image with the point spread function, the control system including a processor;
generating an outlier mask with the control system using an absolute difference array generated by comparing the blurry image and the reblurred image; and
generating the spatial mask using the outlier mask with the control system, the spatial mask being an array of individual weights for individual pixels.

7. The method of claim 6 wherein generating the outlier mask includes generating a transformed array by transforming the absolute difference array to have values in the range of zero to one.

8. The method of claim 7 wherein generating the outlier mask includes generating a binary array from one of the transformed array and absolute difference array; and processing the transformed array with the binary array to generate the outlier mask.

9. The method of claim 6 wherein computing a spatial mask includes (i) computing a boundary mask with the control system, (ii) computing a highlight mask with the control system, (iii) computing an outlier mask with the control system, and (iv) using the boundary mask, the highlight mask, and the outlier mask to compute the spatial mask with the control system.

10. A method for deblurring a blurry image having a point spread function to provide a latent sharp image, the method comprising:
downsampling the blurry image with a control system to create downsampled blurry image, the downsampled blurry image being in the YCbCr color space and including a luminance channel, a Cb chrominance channel, and a Cr chrominance channel, the control system including a processor;
downsampling the point spread function to create a downsampled point spread function with the control system;
performing a phase one of deconvolution on the luminance channel of the downsampled blurry image using an adaptive number of phase one iterations to generate a phase one image with the control system, wherein the number of phase one iterations will depend on a presence of ringing artifacts in the phase one image;
performing a phase two of deconvolution on the chrominance channels of the downsampled blurry image to generate a Cb phase two image and a Cr phase two image with the control system; and
utilizing the phase one image, and the phase two images to generate the latent sharp image with the control system.

11. The method of claim 10 wherein the performing a phase two of deconvolution includes using an adaptive number of phase two iterations, wherein the number of phase two iterations will depend on a presence of ringing artifacts in the Cb phase two image and a Cr phase two image.

12. The method of claim 10 wherein performing a phase two of deconvolution includes the step of generating an edge mask using the phase one image and using the edge mask during the phase two of deconvolution.

13. The method of claim 10 further comprising performing a quality control test on the latent sharp image with the control system.

14. The method of claim 13 wherein performing a quality control test includes using at least one or a sharpness metric and a ringing metric.

15. A method for deblurring a blurry image having a point spread function to provide a latent sharp image, the method comprising:
downsampling the blurry image to create downsampled blurry image with a control system that includes a processor, the downsampled blurry image being in the YCbCr color space and including a luminance channel, a Cb chrominance channel, and a Cr chrominance channel;
downsampling the point spread function to create a downsampled point spread function with the control system;
performing a phase one of deconvolution on the luminance channel of the downsampled blurry image to generate a phase one image with the control system;
performing a phase two of deconvolution on the chrominance channels of the downsampled blurry image to generate a Cb phase two image and a Cr phase two image with the control system;
upsampling the phase one image to create an upsampled phase one image with the control system;
upsampling the Cb phase two image to create an upsampled Cb phase two image with the control system;
upsampling the Cr phase two image to create an upsampled Cr phase two image with the control system;
performing a phase three of deconvolution on a luminance channel of the blurry image to create a phase three image with the control system; and
blending the upsampled phase one image, the upsampled phase two images, and the phase three image to generate the latent sharp image with the control system.

16. The method of claim 15 wherein the step of performing a phase two of deconvolution includes generating an edge mask using the phase one image and using the edge mask during the phase two of deconvolution.

17. The method of claim 15 wherein performing a phase one of deconvolution includes using an adaptive number of iterations depending on a presence of ringing artifacts in the blurry image.

18. The method of claim 15 wherein performing a phase three of deconvolution includes using the upsampled phase one image during the phase three of deconvolution.

19. The method of claim 15 wherein performing a phase one of deconvolution includes using phase one regularization weights and the performing a phase three of deconvolution includes using phase three regularization weights, and wherein the phase three regularization weights are lower than the phase one regularization weights.

20. The method of claim 15 wherein blending includes blending a detail layer of the phase three image with a base layer of the upsampled phase one image to create a blended luminance channel image.

21. The method of claim 20 wherein upsampling the Cb phase two image includes using the blended luminance channel image as a

Assignees

• Nikon Corp

Inventors

• Tezaur Radka
• Ali Gazi
• Kamata Tetsuji

Classifications

• G06T5/004

  Physics · mapped topic

• G06T5/20 Primary

  using local operators · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T5/75

  Unsharp masking · CPC title