Systems and methods for burst image deblurring

Patent metadata

Field	Value
Publication number	US-9998666-B2
Application number	US-201615248895-A
Country	US
Kind code	B2
Filing date	Aug 26, 2016
Priority date	Aug 26, 2015
Publication date	Jun 12, 2018
Grant date	Jun 12, 2018

Abstract

System and Method for automatically removing blur and noise in a plurality of digital images. The system comprises an electronic processor configured to receive the plurality of digital images, perform motion estimation and motion compensation to align the plurality of digital images, determine an alignment of the plurality of digital images with respect to a reference frame, generate a consistency map based on the alignment of the plurality of digital images with respect to the reference frame, combine the plurality of digital images aligned with respect to the reference frame in the Fourier domain using a quality of alignment information from the consistency map to generate an aggregated frame, and apply a post-processing filter to enhance the quality of the aggregated frame.

First claim

What is claimed is:

1. A method for automatically removing blur and noise in a plurality of digital images comprising:
receiving, with an electronic processor, the plurality of digital images including sequential burst images of an object;
performing, with the electronic processor, a motion estimation and a motion compensation to align the plurality of digital images;
determining, with the electronic processor, an alignment of the plurality of digital images with respect to a reference frame;
generating, with the electronic processor, a consistency map based on the alignment of the plurality of digital images with respect to the reference frame;
combining, with the electronic processor, the plurality of digital images aligned with respect to the reference frame in the Fourier domain using a quality of alignment information from the consistency map to generate an aggregated frame; and
applying, with the electronic processor, a post-processing filter to enhance the quality of the aggregated frame.

2. The method of claim 1 , wherein the plurality of digital images is received from a video sequence captured in a digital camera.

3. The method of claim 1 , wherein the plurality of digital images includes an image burst sequence captured from a burst mode of a digital camera.

4. The method of claim 1 , wherein performing, with the electronic processor, the motion estimation includes computing a global homography between each frame and the reference frame.

5. The method of claim 1 , wherein performing, with the electronic processor, the motion estimation includes computing at least one of a homography or an affine transform between each frame and the reference frame.

6. The method of claim 1 , wherein performing, with the electronic processor, the motion estimation includes computing at least one of a homography and an affine transform between a block of a frame and a block of the reference frame.

7. The method of claim 1 , wherein performing with the electronic processor, the motion estimation includes computing optical flow vectors.

8. The method of claim 1 , wherein the consistency map is generated by measuring the difference in color between a block of pixels in one of the aligned frames and the reference frame.

9. The method of claim 1 , wherein the consistency map is generated by measuring the difference between forward and backward optical flow.

10. The method of claim 1 , wherein the aggregated frame is generated by computing
a new set of aligned images with respect to the reference where each frame is transported using the motion compensation and the consistency map.

11. The method of claim 10 , further comprising:
computing Fourier transforms of the new set of aligned images,
computing a weighted average in the Fourier domain of the Fourier transforms of the new set of aligned images, and
computing inverse Fourier transform of the computed weighted average.

12. The method of claim 11 , wherein the weighted average in the Fourier domain is computed locally according to the expression:

P
i

=

ℱ

-
1

⁢

(

∑

i
=

-
M

M

⁢

w
i

·

P
^

i
l

)

,

where the weights w i is large if the magnitude of the Fourier transform of a block of pixels P i is relatively large with respect to the others P j with j different than i.

13. The method of claim 12 , where the weights are computed according, according to

w
i

=



P
^

_

i
l



p

∑

j
=

-
M

M

⁢



P
^

_

j
l



p

where p is a parameter that can vary between 0 and infinite and
| {circumflex over ( P )} i l |=G σ |{circumflex over (P)} i l |,

is a Gaussian smoothed version of the Fourier transform magnitude of a block P i .

14. The method of claim 1 , wherein applying the post-processing filter on the aggregated frame includes performing an unsharp masking.

15. An image processing system comprising:
an electronic processor configured to
receive a plurality of digital images including sequential burst images of an object;
perform motion estimation and a motion compensation to align the plurality of digital images;
determine an alignment of the plurality of digital images with respect to a reference frame;
generate a consistency map based on the alignment of the plurality of digital images with respect to the reference frame;
combine the plurality of digital images aligned with respect to the reference frame in the Fourier domain using a quality of alignment information from the consistency map to generate an aggregated frame; and
apply a post-processing filter to enhance the quality of the aggregated frame thereby removing blur and noise in the plurality of digital images.

16. The image processing system of claim 15 , wherein the aggregated frame is generated by computing a new set of aligned images with respect to the reference where each frame is transported using the motion compensation and the consistency map.

17. The image proc

Assignees

• Univ Duke

Inventors

• Sapiro Guillermo
• Delbracio Mauricio

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20201

  Motion blur correction · CPC title

• H04N23/6811

  based on the image signal · CPC title

• H04N23/667

  Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title