Apparatus and method for generating high dynamic range image from which ghost blur is removed using multi-exposure fusion

Patent metadata

Field	Value
Publication number	US-8989484-B2
Application number	US-201213398323-A
Country	US
Kind code	B2
Filing date	Feb 16, 2012
Priority date	Jul 28, 2011
Publication date	Mar 24, 2015
Grant date	Mar 24, 2015

Abstract

An apparatus and method for generating a High Dynamic Range (HDR) image from which a ghost blur is removed based on a multi-exposure fusion. The apparatus may include an HDR weight map calculation unit to calculate an HDR weight map for multiple exposure frames that are received, a ghost probability calculation unit to calculate a ghost probability for each image by verifying a ghost blur for the multiple exposure frames, an HDR weight map updating unit to update the calculated HDR weight map based on the calculated ghost probability, and a multi-scale blending unit to generate an HDR image by reflecting the updated HDR weight map to the multiple exposure frames.

First claim

What is claimed is:

1. An apparatus for generating a High Dynamic Range (HDR) image from which a ghost blur is removed based on a multi-exposure fusion, the apparatus comprising:
an HDR weight map calculation unit, including at least one processing device, to calculate an HDR weight map for received multiple exposure frames;
a ghost probability calculation unit to calculate a ghost probability for each image based on a difference in a brightness or a gradient between a reference frame and at least one frame among the multiple exposure frames, at least one brightness of the at least one frame being matched;
an HDR weight map updating unit to update the calculated HDR weight map based on the calculated ghost probability; and
a multi-scale blending unit to generate an HDR image by reflecting the updated HDR weight map to the multiple exposure frames,
wherein the HDR weight map updating unit updates the calculated HDR weight map by applying pixels of the calculated ghost probability to respective pixels of the calculated HDR weight map, such that ghost probabilities are uniquely calculated per pixel, and
wherein the ghost probability calculation unit calculates the ghost probability based on a ratio of the difference in the brightness to both a noise level and a threshold control constant.

2. The apparatus of claim 1 , further comprising:
a brightness matching unit to determine the reference frame among the multiple exposure frames, and to match at least one brightness of at least one frame among the multiple exposure frames, based on the determined reference frame.

3. The apparatus of claim 2 , wherein the brightness matching unit determines one of the multiple exposure frames as the reference frame, changes brightnesses of frames other than the reference frame, based on the exposure level of the reference frame, and matches the brightnesses of the other frames with a brightness of the reference frame.

4. The apparatus of claim 2 , wherein the ghost probability calculation unit verifies the ghost blur for the multiple exposure frames, based on the matched brightness of the at least one frame.

5. The apparatus of claim 4 , wherein the ghost probability calculation unit verifies a brightness difference value between the reference frame and each of the at least one frame, based on a result of the matching, and determines that the ghost blur occurs when the brightness difference value is equal to or greater than a threshold.

6. The apparatus of claim 1 , wherein the HDR weight map updating unit modifies an HDR weight of a pixel with the calculated ghost probability that is equal to or greater than a threshold, to be lower than an HDR weight of a pixel with the calculated ghost probability that is equal to or less than the threshold, and updates the HDR weight map.

7. The apparatus of claim 6 , wherein the HDR weight map stores, as an HDR weight, at least one of a contrast, a color saturation, and a well-exposedness in association with a pixel of each of the multiple exposure frames.

8. The apparatus of claim 1 , wherein the multi-scale blending unit receives the multiple exposure frames and the HDR weight map, blends the multiple exposure frames with the HDR weight map using a Gaussian pyramid and a Laplacian pyramid, and generates the HDR image.

9. A method for generating a High Dynamic Range (HDR) image from which a ghost blur is removed based on a multi-exposure fusion, the method comprising:
calculating, by an HDR weight map calculation unit, an HDR weight map for received multiple exposure frames;
calculating, by a ghost probability calculation unit, a ghost probability for each image based on a difference in a brightness or a gradient between a reference frame and at least one frame among the multiple exposure frames, at least one brightness of the at least one frame being matched;
updating, by an HDR weight map updating unit, the calculated HDR weight map based on the calculated ghost probability; and
generating, by a multi-scale blending unit, an HDR image by reflecting the updated HDR weight map to the multiple exposure frames,
wherein the updating comprises updating the calculated HDR weight map by applying pixels of the calculated ghost probability to respective pixels of the calculated HDR weight map, such that ghost probabilities are uniquely calculated per pixel, and
wherein the ghost probability calculation unit calculates the ghost probability based on a ratio of the difference in the brightness to both a noise level and a threshold control constant.

10. The method of claim 9 , further comprising:
determining, by a brightness matching unit, the reference frame among the multiple exposure frames, and matching at least one brightness of at least one frame, among the multiple exposure frames, based on the determined reference frame.

11. The method of claim 10 , further comprising determining one of the multiple exposure frames as the reference frame, changing brightnesses of frames other than the reference frame, based on an exposure level of the reference frame, and matching the brightnesses of the other frames with a brightness of the reference frame.

12. The method of claim 10 , further comprising verifying the ghost blur for the multiple exposure frames, based on the matched brightness of the at least one frame.

13. The method of claim 12 , further comprising verifying a brightness difference value between the reference frame and each of the at least one frame, based on a result of the matching, and determining that the ghost blur occurs when the brightness difference value is equal to or greater than a threshold.

14. The method of claim 9 , wherein the updating comprises modifying an HDR weight of a pixel with the calculated ghost probability that is equal to or greater than a threshold, to be lower than an HDR weight of a pixel with the calculated ghost probability that is equal to or less than the threshold, and updating the HDR weight map.

15. The method of claim 14 , wherein the HDR weight map stores, as an HDR weight, at least one of a contrast, a color saturation, and a well-exposedness in association with a pixel of each of the multiple exposure frames.

16. The method of claim 9 , wherein the generating comprises receiving the multiple exposure frames and the HDR weight map, blending the multiple exposure frames and the HDR weight map using a Gaussian pyramid and a Laplacian pyramid, and generating the HDR image.

17. The apparatus of claim 1 , wherein the HDR weight map updating unit controls a weighted sum in pixels of each frame to be “1” through a series of normalizations.

18. The apparatus of claim 1 , wherein the ghost blur is removed by determining, as a motion detection characteristic, a deviation between original exposure frames with different brightnesses and exposure frames with a matched brightness.

Assignees

• Moon Young Su
• Tai Yong Min
• Cho Jung Uk
• Lee Shi Hwa
• Samsung Electronics Co Ltd

Inventors

• Moon Young Su
• Tai Yong Min
• Cho Jung Uk
• Lee Shi Hwa

Classifications

• H04N23/741 Primary

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• G06T2207/20076

  Probabilistic image processing · CPC title

• G06T2207/10144

  Varying exposure · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title