System and method for processing input images before generating a high dynamic range image

Patent metadata

Field	Value
Publication number	US-9916646-B2
Application number	US-201615245402-A
Country	US
Kind code	B2
Filing date	Aug 24, 2016
Priority date	Feb 24, 2014
Publication date	Mar 13, 2018
Grant date	Mar 13, 2018

Abstract

An image processing system for generating a high dynamic range image from a first image and a second image is presented. The system comprises a first displacement estimator adapted to estimate a displacement of image content between a reference image derived from the first image and an image to be warped derived from the second image. The system comprises a first warping unit that compensates the estimated displacement by warping the image to be warped resulting in a first warped image. The system comprises a first error detector adapted to detect geometric warping errors within the first warped image and a first error compensator adapted to compensate the estimated geometric warping errors within the first warped image resulting in a first error corrected image. The system comprises a high dynamic range calculator adapted to calculate the high-dynamic range image based upon the first error corrected image and the reference image.

First claim

What is claimed is:

1. An image processing system for generating a high dynamic range image from a first image and a second image, comprising:
a memory comprising instructions; and
a processor coupled to the memory, wherein the instructions cause the processor to be configured to:
estimate a displacement of image content between a reference image derived from the first image and an image to be warped derived from the second image;
compensate the estimated displacement by warping the image to be warped to generate a first warped image;
compute intensity differences between each pixel of the first warped image to a corresponding pixel of the reference image;
compute a histogram of the computed intensity differences;
adaptively determine an error threshold by analyzing gradients of consecutive values of the histogram and selecting the error threshold as a value where the gradient is maximal;
detect geometric warping errors within the first warped image based on the error threshold;
compensate the detected geometric warping errors in the first warped image resulting in a first error corrected image; and
calculate the high dynamic range image based upon the first error corrected image and the reference image.

2. The image processing system according to claim 1 , wherein the instructions further cause the processor to be configured to estimate the displacement of the image content by at least one of:
estimating an optical flow;
performing a block based search; and
performing a patch matching.

3. The image processing system according to claim 1 , wherein the instructions further cause the processor to be configured to:
determine the error threshold from the histogram; and
determine geometric warping errors by comparing the intensity differences to the error threshold resulting in a binary map of geometric warping error pixels.

4. The image processing system according to claim 3 , wherein the instructions further cause the processor to be configured to determine the error threshold so that difference values below the error threshold indicate photometric warping errors and values above the error threshold indicate geometric warping errors.

5. The image processing system according to claim 3 , wherein the instructions further cause the processor to be configured to:
compute the intensity differences independently for different color channels;
compare the intensity differences of the different color channels to the error threshold; and
determine a geometric warping error of a pixel when at least one of the intensity differences of the different color channels is higher than the error threshold, or when intensity differences of all color channels are higher than the error threshold.

6. The image processing system according to claim 3 , wherein the instructions further cause the processor to be configured to:
compute the intensity differences independently for different color channels;
compute independent histograms for the different color channels;
determine an independent error threshold for each color channel;
compare the intensity differences of the color channels to corresponding error thresholds; and
determine a warping error of a pixel when at least one of the intensity differences of the color channels is higher than the corresponding error threshold, or when intensity differences of all color channels are higher than the corresponding error thresholds.

7. The image processing system according to claim 3 , wherein the instructions further cause the processor to be configured to:
set search windows of predetermined or user-defined size around each pixel of the first warped image, for which a geometric warping error was detected;
determine at which pixels within each search window no geometric warping error was detected resulting in valid pixels for each search window;
calculate correction values for the pixels of the first warped image, at which geometric warping errors were detected, each as an average of intensities of at least some of the valid pixels of the search window; and
replace the pixels of the first warped image, at which geometric warping errors were detected, by corresponding calculated correction values resulting in the first error corrected image.

8. The image processing system according to claim 7 , wherein the instructions further cause the processor to be configured to calculate the correction values each as an average of intensities of only valid pixels, whose intensity value difference to the intensity value of a presently corrected pixel are below a predetermined or user-selectable correction threshold.

9. The image processing system according to claim 7 , wherein the instructions further cause the processor to be configured to;
compute a weighted map based on the binary map of geometric warping errors and intensity gradients of the first warped image, wherein a lower weight is allocated to pixels of high gradient value, such as edges and boundaries, and wherein a higher weight is allocated to pixels of low gradient value, resulting in a weighted map; and
weight the valid pixels of the search windows using the weighted map before calculating the correction values.

10. The image processing system according to claim 1 , wherein the instructions further cause the processor to be configured to compensate different exposure times of the first image and the second image by changing at least one of a brightness and saturation of the first image or of the second image resulting in the reference image and the image to be warped.

11. The image processing system according to claim 1 , wherein the instructions further cause the processor to be configured to:
estimate displacement of image content between the reference image and a further image to be warped derived from a third image;
compensate the estimated displacement by warping the further image to be warped resulting in a second warped image;
detect geometric warping errors within the second warped image;
compensate the detected geometric warping errors in the second warped image resulting in a second error corrected image; and
calculate the high dynamic range image based on the first error corrected image, the second error corrected image and the reference image.

12. A method for generating a high dynamic range image from a first image and a second image, comprising:
estimating with a processor, displacement of image content between a reference image derived from the first image and an image to be warped derived from the second image;
compensating, with the processor, the estimated displacement by warping the image to be warped to generate a first warped image;
computing, with the processor, intensity differences between each pixel of the first warped image to a corresponding pixel of the reference image;
computing, with the processor, a histogram of the computed intensity differences;
adaptively determining, with the processor, an error threshold by analyzing gradients of consecutive values of the histogram and selecting the error threshold as a value where the gradient is maximal;
detecting, with the processor, geometric warping errors within the first warped image based on the error threshold;
compensating, with the processor, the detected geometric warping errors in the first warped image to generate a first error corrected image; and
calculating, with the processor, the high dynamic range image based on the first error corrected image and the reference image.

13. A non-transitory computer readable medium including at least one computer program code stored therein for generating a high dynamic range image associated with a computing

Assignees

• Huawei Tech Co Ltd

Inventors

• Bouzaraa Fahd
• Cordara Giovanni
• Urfalioglu Onay

Classifications

• G06T2207/20216

  Image averaging · CPC title

• G06T2207/10004

  Still image; Photographic image · CPC title

• G06T2207/20208

  High dynamic range [HDR] image processing · CPC title

• G06T5/007 Primary

  Physics · mapped topic

• G06T5/40

  using histogram techniques · CPC title