Image fusion processing module

Patent metadata

Field	Value
Publication number	US-10853928-B2
Application number	US-201916370662-A
Country	US
Kind code	B2
Filing date	Mar 29, 2019
Priority date	Mar 29, 2019
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

Embodiments relate to fusion processing between two images captured with two different exposure times to generate a fused image with a higher dynamic range. An unscaled single color version of a first image is blended with another unscaled single color version of a second image to generate an unscaled single color version of the fused image. A downscaled multi-color version of the first image is blended with a downscaled multi-color version of the second image to generate a downscaled multi-color version of the fused image of a plurality of downscaled versions of the fused image. A first downscaled multi-color version of the fused image is generated by upscaling and accumulating the plurality of downscaled versions of the fused image. The first downscaled multi-color version of the fused image has a pixel resolution lower than a pixel resolution of the unscaled single color version of the fused image.

First claim

What is claimed is:

1. An apparatus for image fusion processing, comprising:
an image fusion circuit configured to:
determine a blend parameter for each pixel in at least a portion of a first image by using pixel values in a first patch of an unsealed single color version of the first image and pixel values in a second patch of another unsealed single color version of a second image, the first and second images capturing a same scene with different exposure times, and
blend, per pixel, the unsealed single color version with the other unsealed single color version using the blend parameter for each pixel to generate an unsealed single color version of a fused image; and

a multi-scale image fusion circuit configured to:
blend, per pixel, a downscaled multi-color version of the first image with a downscaled multi-color version of the second image to generate a downscaled multi-color version of the fused image of a plurality of downscaled versions of the fused image, and
generate a first downscaled version of the fused image by upscaling and accumulating the plurality of downscaled versions of the fused image, the first downscaled version comprising a plurality of color components and having a pixel resolution lower than a pixel resolution of the unsealed single color version of the fused image.

2. The apparatus of claim 1 , wherein the image fusion circuit is further configured to:
blend, for each pixel, the unsealed single color version with the other unsealed single color version by processing the first patch of the unsealed single color version and the second patch of the other unsealed single color version.

3. The apparatus of claim 1 , wherein the image fusion circuit further comprising:
a calculator circuit configured to:
determine a patch distance for each pixel by processing photometric distances between pixels in the first patch and corresponding pixels in the second patch,
determine a cross-correlation value for each pixel by determining a cross variance between pixel values of the first patch and pixel values of the second patch, and
determine the blend parameter for each pixel as a function of the patch distance for each pixel and the cross-correlation value for each pixel; and

a blending circuit configured to blend, per pixel, the unsealed single color version with the other unsealed single color version using the blend parameter for each pixel to generate the unsealed single color version of the fused image.

4. The apparatus of claim 1 , wherein the multi-scale image fusion circuit is further configured to:
blend, per pixel, the downscaled multi-color version of the first image with the downscaled multi-color version of the second image by processing a first patch of the downscaled multi-color version of the first image and a second patch of the downscaled multi-color version of the second image.

5. The apparatus of claim 4 , wherein the multi-scale image fusion circuit further comprising:
an upscaling circuit configured to generate a plurality of downscaled multi-color versions of the first image and a plurality of downscaled multi-color versions of the second image by sequentially upscaling corresponding downscaled multi-color versions of the first and second images;
a calculator circuit configured to:
determine a patch distance for each pixel by processing photometric distances between pixels in the first patch of the downscaled multi-color version of the first image and corresponding pixels in the second patch of the downscaled multi-color version of the second image,
determine a cross-correlation value for each pixel by determining a cross variance between pixel values of the first patch of the downscaled multi-color version of the first image and pixel values of the second patch of the downscaled multi-color version of the second image, and
determine another blend parameter for each pixel as a function of the patch distance for each pixel and the cross-correlation value for each pixel; and

a blending circuit configured to blend, per pixel, the downscaled multi-color version of the first image with the downscaled multi-color version of the second image using the other blend parameter for each pixel to generate the downscaled multi-color version of the fused image.

6. The apparatus of claim 5 , wherein the calculator circuit is further configured to:
determine the patch distance for each pixel by processing the photometric distances between the pixels in the first patch of a high frequency component of the downscaled multi-color version of the first image and the pixels in the second patch of a high frequency component of the downscaled multi-color version of the second image; and
determine the cross-correlation value for each pixel by determining the cross variance between the first patch of a low frequency component of the downscaled multi-color version of the first image and the second patch of a low frequency component of the downscaled multi-color version of the second image.

7. The apparatus of claim 5 , wherein the calculator circuit is further configured to:
determine a normalized version of the cross-correlation value for each pixel by normalizing the cross variance using a variance of the pixel values of the first patch of the downscaled multi-color version of the first image and a variance of the pixel values of the second patch of the downscaled multi-color version of the second image; and
determine the other blend parameter for each pixel as a function of the patch distance for each pixel and the normalized version of the cross-correlation value for each pixel.

8. The apparatus of claim 1 , wherein the first downscaled version of the fused image has the pixel resolution equal to a quarter of the pixel resolution of the unsealed single color version of the fused image.

9. The apparatus of claim 1 , wherein the apparatus further comprising:
a sub-band splitter (SBS) circuit configured to perform sub-band splitting operations on the unsealed single color version of the fused image and on the first downscaled version of the fused image.

10. A method of image fusion processing, comprising:
determining a blend parameter for each pixel in at least a portion of a first image by using pixel values in a first patch of an unsealed single color version of the first image and pixel values in a second patch of another unsealed single color version of a second image, the first and second images capturing a same scene with different exposure times;
blending, per pixel, the unsealed single color version with the other unsealed single color version using the blend parameter for each pixel to generate an unscaled single color version of a fused image;
blending, per pixel, a downscaled multi-color version of the first image with a downscaled multi-color version of the second image to generate a downscaled multi-color version of the fused image of a plurality of downscaled versions of the fused image; and
generating a first downscaled version of the fused image by upscaling and accumulating the plurality of downscaled versions of the fused image, the first downscaled version comprising a plurality of color components and having a pixel resolution lower than a pixel resolution of the unscaled single color version of the fused image.

11. The method of claim 10 , further comprising:
blending, for each pixel, the unscaled single color version with the other unscaled single color version by processing the first patch of the unscaled single color version and the second patch of the other unscaled single color version.

12. The method of claim 10 , further comprising:
determining a patch distance for each pixel by processing photometric d

Assignees

• Apple Inc

Inventors

• Smirnov Maxim
• Lamburn Elena

Classifications

• H04N23/743

  Bracketing, i.e. taking a series of images with varying exposure conditions · CPC title

• H04N23/72

  Combination of two or more compensation controls · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N25/60

  Noise processing, e.g. detecting, correcting, reducing or removing noise · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title