Method and system for deblurring a blurred image

Patent metadata

Field	Value
Publication number	US-12229926-B2
Application number	US-202217863501-A
Country	US
Kind code	B2
Filing date	Jul 13, 2022
Priority date	Jul 13, 2022
Publication date	Feb 18, 2025
Grant date	Feb 18, 2025

Abstract

A method for deblurring a blurred image includes dividing the blurred image into overlapping regions each having a size and an offset from neighboring overlapping regions along a first direction as determined by a period of a ringing artifact in the blurred image, or by obtained blur characteristics relating to the blurred image and/or attributable to the optical system, or by a detected cause capable of producing the blur characteristics, stacking the overlapping regions to produce a stacked output, wherein the overlapping regions are sequentially organized along the first direction, convolving the stacked output through a first convolutional neural network (CNN) to produce a first CNN output having reduced blur as compared to the stacked output, and assembling the first CNN output into a re-assembled image, and processing the re-assembled image through a second CNN to produce a deblurred image having reduced residual artifacts as compared to the re-assembled image.

First claim

What is claimed is:

1. A method for deblurring a blurred image captured by an optical system, comprising:
dividing the blurred image into a plurality of overlapping regions each having a size and an offset from neighboring overlapping regions along a first direction, wherein the size and offset are determined by a period of a ringing artifact in the blurred image if the blurred image has been filtered, or by obtained blur characteristics relating to the blurred image and/or attributable to the optical system, or by a detected cause capable of producing the blur characteristics;
stacking the plurality of overlapping regions to produce a stacked output, wherein the overlapping regions are sequentially organized along the first direction;
convolving the stacked output through a first convolutional neural network (CNN) to produce a first CNN output having reduced blur as compared to the stacked output; and
assembling the first CNN output into a re-assembled image.

2. The method of claim 1 , wherein the offset between neighboring overlapping regions is approximately equal to the period of the ringing artifact or a parameter derived from the blur characteristics, and wherein the size of each overlapping region along the first direction is approximately equal to or greater than the period of the ringing artifact or the parameter derived from the blur characteristics.

3. The method of claim 1 , wherein each of the overlapping regions in the stacked output is processed in a respective separate channel in the first CNN.

4. The method of claim 1 , further comprising:
processing the re-assembled image through a second CNN to produce the deblurred image, wherein the deblurred image has reduced residual artifacts as compared to the re-assembled image.

5. The method of claim 4 , wherein the first CNN has a first number of channels and the second CNN has a second number of channels that is different from the first number of channels.

6. The method of claim 4 , wherein one or both of the first and second CNNs utilize a U-net CNN structure.

7. The method of claim 1 , wherein the residual artifacts are produced by the step of convolving the stacked output through the first CNN and/or by the step of stacking the plurality of overlapping regions to produce the stacked output.

8. The method of claim 1 , further comprising:
filtering the blurred image through a Tikhonov filter, a Wiener filter or another filter before the dividing step.

9. The method of claim 1 , further comprising:
obtaining (i) the blur characteristics relating to the blurred image and/or attributable to the optical system and/or (ii) the cause capable of producing the blur characteristics.

10. The method of claim 1 , wherein the dividing step is performed by a dividing module, the stacking step is performed by a stacking module, the convolving step is performed by a convolving module, and the assembling step is performed by an assembling module;
wherein the dividing module, the stacking module, the convolving module and the assembling module are operatively connected to form a deblurring CNN module.

11. The method of claim 10 , wherein the filtering step is performed by a filtering module and the processing step is performed by a processing module, wherein the deblurring CNN module further includes the filtering module and the processing module.

12. The method of claim 10 , further comprising:
training the deblurring CNN module by supervised training, wherein the supervised training includes:
providing a database of non-blurred stock images;
generating a synthetically blurred stock image corresponding to each non-blurred stock image using a blurring scheme;
producing a deblurred stock image corresponding to each synthetically blurred stock image using the deblurring CNN module and a weighting scheme;
comparing each deblurred stock image with the corresponding non-blurred stock image to determine a difference therebetween; and
repeating the generating, producing and comparing steps utilizing varied blurring schemes and weighting schemes until the difference is minimized; or

training the deblurring CNN module by discriminative training in a generative adversarial network (GAN), wherein the GAN includes a generator which provides one or more candidate data to a discriminator and wherein the discriminator receives the candidate data and domain data, wherein the discriminative training includes:
providing the database of non-blurred stock images as the domain data;
using the deblurring CNN module as the generator and supplying one or more deblurred images as the candidate data to the discriminator; and
utilizing a deep neural network (DNN) as the discriminator, receiving the one or more deblurred images as the candidate data and the non-blurred stock images as the domain data, and discriminating between the candidate data and the domain data to produce discriminations therebetween;
wherein the generator and the discriminator compete with each other in a zero-sum game of making adjustments to either the generator or the discriminator based on the discriminations until a predetermined degree of convergence is achieved between the generator and the discriminator.

13. A method for deblurring a blurred image captured by an optical system, comprising:
obtaining, by an obtaining module, blur characteristics relating to the blurred image and/or attributable to the optical system, and/or a cause capable of producing the blur characteristics;
filtering, by a filtering module, the blurred image through a filter to produce a filtered image;
dividing, by a dividing module, the filtered image into a plurality of overlapping regions each having a size and an offset from neighboring overlapping regions along a first direction, wherein the size and offset are determined by a period of a ringing artifact in the filtered image if the blurred image has been filtered, or by the obtained blur characteristics, or by the detected cause;
stacking, by a stacking module, the plurality of overlapping regions to produce a stacked output, wherein the overlapping regions are sequentially organized along the first direction;
convolving, by a convolving module, the stacked output through a first U-net convolutional neural network (CNN) to produce a first CNN output having reduced blur as compared to the stacked output;
assembling, by an assembling module, the first CNN output into a re-assembled image; and
processing, by a processing module, the re-assembled image through a second U-net CNN to produce a deblurred image having reduced residual artifacts as compared to the re-assembled image;
wherein the filtering module, the dividing module, the stacking module, the convolving module, the assembling module, and the processing module are operatively connected to form a deblurring CNN module.

14. The method of claim 13 , further comprising:
training the deblurring CNN module by supervised training, wherein the supervised training includes:
providing a database of non-blurred stock images;
generating a synthetically blurred stock image corresponding to each non-blurred stock image using a blurring scheme;
producing a deblurred stock image corresponding to each synthetically blurred stock image using the deblurring CNN module and a weighting scheme;
comparing each deblurred stock image with the corresponding non-blurred stock image to determine a difference therebetween; and
repeating the generating, producing and comparing steps utilizing varied blurring schemes and weighting schemes until the difference is minimized; or

training the deblurring CNN module by discriminative training in a

Assignees

• Gm Global Tech Operations Llc

Inventors

• Philipp Tzvi
• Levi Dan
• Silberstein Shai

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20201

  Motion blur correction · CPC title

• G06T2207/10048

  Infrared image · CPC title

• G06T5/20

  using local operators · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title