Restoring images using deconvolution

Patent metadata

Field	Value
Publication number	US-12482075-B2
Application number	US-202217993357-A
Country	US
Kind code	B2
Filing date	Nov 23, 2022
Priority date	Jun 8, 2022
Publication date	Nov 25, 2025
Grant date	Nov 25, 2025

Abstract

In one embodiment, a method includes generating, from an accessed image, one or more image patches, and for each image patch: (1) accessing a set of point-spread functions (PSFs), wherein each PSF in the set of PSFs corresponds to one of a plurality of points in the image patch; (2) generating a set of deconvolved image patches by deconvolving the image patch with each PSF from the set of PSFs for that patch; (3) determining, for each of one or more portions of the image patch, a set of weights, wherein each weight in the set of weights is associated with one of the deconvolved image patches; and (4) generating a restored image patch by interpolating the set of deconvolved image patches based on the set of weights.

First claim

What is claimed is:

1 . A method comprising:
generating, from an accessed blurred image, a plurality of blurred image patches; and for each blurred image patch:
accessing a set of point-spread functions (PSFs), wherein each PSF in the set of PSFs corresponds to a different one of a plurality of points in the blurred image patch;
generating, for the blurred image patch, a set of multiple deconvolved image patches, each deconvolved image patch generated by deconvolving the blurred image patch with a different one of the PSFs from the set of PSFs for that blurred image patch;
determining, for each of one or more portions of the blurred image patch, a set of weights, wherein each weight in the set of weights is associated with a different one of the multiple deconvolved image patches; and
generating a restored image patch for the blurred image patch by interpolating the set of multiple deconvolved image patches based on the set of weights; and
generating a deblurred image by replacing each blurred image patch with its corresponding restored image patch and then stitching the image patches together.

2 . The method of claim 1 , wherein:
the accessed blurred image comprises a first channelized image, and the method further comprises generating the channelized image by:
accessing a color image; and
separating the color image into a plurality of channelized images, each channelized image corresponding to a color channel.

3 . The method of claim 2 , further comprising repeating the method of claim 1 for each of the plurality of channelized images.

4 . The method of claim 3 , further comprising:
for each of the plurality of channelized images, generating a restored channelized image based on the plurality of blurred image patches for the blurred image; and generating a restored, deblurred color image by combining each of the restored channelized images.

5 . The method of claim 1 , wherein:
the accessed blurred image comprises an extended image; and the method further comprises generating the extended image by:
accessing an unextended image; and
generating the extended image by reflecting at least a portion of the unextended image.

6 . The method of claim 1 , wherein each blurred image patch includes a region of interest that is smaller than the blurred image patch, and wherein the region of interest comprises the one or more portions of the blurred image patch.

7 . The method of claim 1 , wherein for each blurred image patch at least some of the PSFs in the set of PSFs for that patch are also in a different set of PSFs for a different, overlapping blurred image patch.

8 . The method of claim 1 , wherein each of the one or more portions of the blurred image patch comprises a pixel.

9 . The method of claim 1 , wherein determining the set of weights comprises determining a distance from the portion of the blurred image patch to a boundary that is defined at least in part by the set of PSFs.

10 . The method of claim 9 , wherein each blurred image patch is a rectangular image patch, and wherein each vertex of the rectangular image patch is a point that corresponds to one of the PSFs in the set of PSFs.

11 . The method of claim 9 , wherein generating the restored image patch comprises, for each of the one or more portions of the image patch, determining a final output value for that portion according to a rectangular interpolation I.

12 . The method of claim 1 , wherein:
the accessed blurred image corresponds to an image captured by a sensor of a computing device that is mounted behind a display of the computing device; and each PSF in the set of PSFs deblurs at least some distortion in the image caused by the display.

13 . The method of claim 12 , wherein at least one PSF in the set of PSFs is determined based on computer simulation.

14 . The method of claim 13 , further comprising:
accessing a display mask associated with the display; and performing a spatial transformation on the display mask to obtain a transformed display mask.

15 . The method of claim 14 , further comprising:
accessing an aperture mask associated with a lens of the sensor; and performing a spatial transformation on the aperture mask to obtain a transformed aperture mask.

16 . The method of claim 15 , further comprising generating the at least one PSF by simulating:
generating a spherical wave at an object plane; generating a light field based on an interaction between the spherical wave and the transformed display mask; modifying the light field based on an interaction between the light field and the transformed aperture mask; and determining a response of a sensor to the modified light field.

17 . One or more non-transitory computer readable storage media embodying instructions and coupled to one or more processors that are operable to execute the instructions to:
generate, from an accessed blurred image, a plurality of blurred image patches; and for each image patch:
access a set of point-spread functions (PSFs), wherein each PSF in the set of PSFs corresponds to a different one of a plurality of points in the blurred image patch;
generate, for the blurred image patch, a set of multiple deconvolved image patches, each deconvolved image patch generated by deconvolving the blurred image patch with a different one of the PSFs from the set of PSFs for that blurred image patch;
determine, for each of one or more portions of the blurred image patch, a set of weights, wherein each weight in the set of weights is associated with a different one of the multiple deconvolved image patches; and
generate a restored image patch for the blurred image patch by interpolating the set of multiple deconvolved image patches based on the set of weights.

18 . The media of claim 17 , wherein determining the set of weights comprises determining a distance from the portion of the blurred image patch to a boundary that is defined at least in part by the set of PSFs.

19 . A system comprising one or more processors and a non-transitory computer readable storage media embodying instructions coupled to the one or more processors, the one or more processors operable to execute the instructions to:
generate, from an accessed image, one or more image patches; and for each image patch:
access a set of point-spread functions (PSFs), wherein each PSF in the set of PSFs corresponds to a different one of a plurality of points in the blurred image patch;
generate, for the blurred image patch, a set of multiple deconvolved image patches, each deconvolved image patch generated by deconvolving the blurred image patch with a different one of the PSFs from the set of PSFs for that blurred image patch;
determine, for each of one or more portions of the blurred image patch, a set of weights, wherein each weight in the set of weights is associated with a different one of the multiple deconvolved image patches; and
generate a restored image patch for the blurred image patch by interpolating the set of multiple deconvolved image patches based on the set of weights.

20 . The system of claim 19 , wherein determining the set of weights comprises determining a distance from the portion of the blurred image patch to a boundary that is defined at least in part by the set of PSFs.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Liu Changgeng
• Zhao Luxi
• Jiang Ziwen
• Abdelhamed Abdelrahman
• Punnappurath Abhijith
• Zhao Ye
• Post Ernest Rehmatulla
• Brown Michael
• Sadi Sajid

Classifications

• G06T2207/20201

  Motion blur correction · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title