Kernel-aware super resolution

Patent metadata

Field	Value
Publication number	US-11615510-B2
Application number	US-202017139885-A
Country	US
Kind code	B2
Filing date	Dec 31, 2020
Priority date	Oct 21, 2020
Publication date	Mar 28, 2023
Grant date	Mar 28, 2023

Abstract

An electronic device includes at least one imaging sensor and at least one processor coupled to the at least one imaging sensor. The at least one imaging sensor is configured to capture a burst of image frames. The at least one processor is configured to generate a low-resolution image from the burst of image frames. The at least one processor is also configured to estimate a blur kernel based on the burst of image frames. The at least one processor is further configured to perform deconvolution on the low-resolution image using the blur kernel to generate a deconvolved image. In addition, the at least one processor is configured to generate a high-resolution image using super resolution (SR) on the deconvolved image.

First claim

What is claimed is:

1. An electronic device configured to provide for kernel-aware super resolution, the electronic device comprising:
at least one imaging sensor configured to capture a burst of image frames; and
at least one processor coupled to the at least one imaging sensor, the at least one processor configured to:
generate a low-resolution image from the burst of image frames;
estimate a blur kernel based on the burst of image frames;
perform deconvolution on the low-resolution image using the blur kernel to generate a deconvolved image;
generate multiple high-resolution images using different prior regularization parameters for super resolution on the deconvolved image; and
blend the high-resolution images to generate a final image of a scene.

2. The electronic device of claim 1 , wherein the at least one processor is further configured to iteratively perform deconvolution and super resolution on the low-resolution image in order to generate the high-resolution images.

3. The electronic device of claim 1 , wherein the at least one processor is further configured to set a noise level to suppress artifacts of the deconvolution.

4. The electronic device of claim 1 , wherein the multiple high-resolution images include a clean super resolution image and a detailed super resolution image.

5. The electronic device of claim 1 , wherein the at least one processor is configured to approximate the blur kernel using a parameterized anisotropic Gaussian function that is consistent with an average blur kernel estimated from the burst of image frames.

6. The electronic device of claim 1 , wherein the at least one processor is further configured to:
generate gradient maps using a bilateral filter, a shock filter, and magnitude thresholding on the low-resolution image; and
estimate an initial blur kernel using an optimization of a cost function that requires fidelity between the gradient maps of the low-resolution image.

7. The electronic device of claim 1 , wherein the at least one processor is further configured to generate an average blur kernel from a plurality of blur kernels estimated from a plurality of patches and approximated by a parameterized anisotropic Gaussian function.

8. A method for kernel-aware super resolution, the method comprising:
capturing, using at least one imaging sensor of an electronic device, a burst of image frames;
generating, using at least one processor of the electronic device, a low-resolution image from the burst of image frames;
estimating, using the at least one processor, a blur kernel based on the burst of image frames;
performing, using the at least one processor, deconvolution on the low-resolution image using the blur kernel to generate a deconvolved image;
generating, using the at least one processor, multiple high-resolution images using different prior regularization parameters for super resolution on the deconvolved image; and
blending the high-resolution images to generate a final image of a scene.

9. The method of claim 8 , further comprising:
iteratively performing deconvolution and super resolution on the low-resolution image in order to generate the high-resolution images.

10. The method of claim 8 , further comprising:
setting a noise level to suppress artifacts of the deconvolution.

11. The method of claim 8 , wherein the multiple high-resolution images include a clean super resolution image and a detailed super resolution image.

12. The method of claim 8 , further comprising:
approximating the blur kernel using a parameterized anisotropic Gaussian function that is consistent with an average blur kernel estimated from the burst of image frames.

13. The method of claim 8 , further comprising:
generating gradient maps using a bilateral filter, a shock filter, and magnitude thresholding on the low-resolution image; and
estimating an initial blur kernel using an optimization of a cost function that requires fidelity between the gradient maps of the low-resolution image.

14. The method of claim 8 , further comprising:
generating an average blur kernel from a plurality of blur kernels estimated from a plurality of patches and approximated by a parameterized anisotropic Gaussian function.

15. A non-transitory machine readable medium storing instructions that are configured to provide for kernel-aware super resolution, wherein the instructions, when executed by at least one processor of an electronic device, cause the at least one processor to:
obtain a burst of image frames;
generate a low-resolution image from the burst of image frames;
estimate a blur kernel based on the burst of image frames;
perform deconvolution on the low-resolution image using the blur kernel to generate a deconvolved image;
generate multiple high-resolution images using different prior regularization parameters for super resolution on the deconvolved image; and
blend the high-resolution images to generate a final image of a scene.

16. The non-transitory machine readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to:
iteratively perform deconvolution and super resolution on the low-resolution image in order to generate the high-resolution images.

17. The non-transitory machine readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to:
set a noise level to suppress artifacts of the deconvolution.

18. The non-transitory machine readable medium of claim 15 , wherein the multiple high-resolution images include a clean super resolution image and a detailed super resolution image.

19. The non-transitory machine readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to:
approximate the blur kernel using a parameterized anisotropic Gaussian function that is consistent with an average blur kernel estimated from the burst of image frames.

20. The non-transitory machine readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to:
generate gradient maps using a bilateral filter, a shock filter, and magnitude thresholding on the low-resolution image; and
estimate an initial blur kernel using an optimization of a cost function that requires fidelity between the gradient maps of the low-resolution image.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Choi Baekdu
• Zhen Ruiwen
• Glotzbach John W
• Sheikh Hamid R

Classifications

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title

• G06T2207/20028

  Bilateral filtering · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title