Multispectral image processing system and method

Patent metadata

Field	Value
Publication number	US-11388355-B2
Application number	US-202016900621-A
Country	US
Kind code	B2
Filing date	Jun 12, 2020
Priority date	Jun 13, 2019
Publication date	Jul 12, 2022
Grant date	Jul 12, 2022

Abstract

Devices, methods, and non-transitory program storage devices are disclosed herein to provide improved multi-spectral image processing techniques for generating an enhanced output image, the techniques comprising: obtaining an N-channel (e.g., multispectral) input image; determining fusion weights and fallback weights (e.g., relative intensity weights) for each of the N-channels of the input image; blending the fusion and fallback weights based on an amount of gradient information to generate blended weights; modulating the blended weights for a plurality of frequency band representations of the input image; applying the modulated blended weights to the corresponding frequency band representations of the input image to generate a plurality of output image frequency band representations; producing an output luma image, based on the plurality of output image frequency band representations; and generating an output RGB image, based on the output luma image, which may then, e.g., be displayed to a user or stored to non-volatile memory.

First claim

What is claimed is:

1. A device, comprising:
a memory;
one or more image capture devices;
a user interface; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain an N-channel input image;
determine fusion weights and fallback weights for each of the N-channels of the input image;
blend the fusion and fallback weights based on an amount of gradient information in the input image to generate blended weights;
modulate the blended weights for a plurality of frequency band representations of the input image;
apply the modulated blended weights to the corresponding frequency band representations of the input image to generate a plurality of output image frequency band representations; and
produce an output luma image, based on the plurality of output image frequency band representations.

2. The device of claim 1 , wherein the N-channel input image comprises an RGB-IR image.

3. The device of claim 1 , wherein the instructions further comprise instructions to:
generate an output RGB image based on the output luma image.

4. The device of claim 1 , wherein the fusion weights are further determined based, at least in part, on one or more of the following:
a principal characteristic vector of an outer product of Jacobian matrices of the input image's N-channels;
a local entropy estimate from an input image channel; or
a gradient magnitude estimate from an input image channel.

5. The device of claim 1 , wherein at least one of the plurality of frequency band representations comprises a high frequency band representation, and wherein the fusion weights are further determined based, at least in part, on information in the high frequency band representation.

6. The device of claim 1 , wherein the frequency band representations of the input image are created by a multiscale decomposition process.

7. The device of claim 1 , wherein the amount of gradient information at a pixel in the input image is determined based, at least in part, on: a size of a largest eigenvalue of a Jacobian matrix of gradients for the input image; and a noise estimate for the pixel.

8. The device of claim 1 , wherein the fallback weight for a given input image channel comprises: a weight based on the input intensity of the given input image channel relative to a summation of the input intensities of the N-channels of the input image.

9. A non-transitory computer readable medium comprising computer readable instructions configured to cause one or more processors to:
obtain an N-channel input image;
determine fusion weights and fallback weights for each of the N-channels of the input image;
blend the fusion and fallback weights based on an amount of gradient information in the input image to generate blended weights;
modulate the blended weights for a plurality of frequency band representations of the input image;
apply the modulated blended weights to the corresponding frequency band representations of the input image to generate a plurality of output image frequency band representations; and
produce an output luma image, based on the plurality of output image frequency band representations.

10. The non-transitory computer readable medium of claim 9 , wherein the N-channel input image comprises an RGB-IR image.

11. The non-transitory computer readable medium of claim 9 , wherein the plurality of frequency band representations comprises a high frequency band representation, and wherein the fusion weights are further determined based, at least in part, on information in the high frequency band representation.

12. The non-transitory computer readable medium of claim 9 , wherein the frequency band representations of the input image are created by a multiscale decomposition process.

13. The non-transitory computer readable medium of claim 9 , wherein the amount of gradient information at a pixel in the input image is determined based, at least in part, on: a size of a largest eigenvalue of a Jacobian matrix of gradients for the input image; and a noise estimate for the pixel.

14. The non-transitory computer readable medium of claim 9 , wherein the fallback weight for a given input image channel comprises: a weight based on the input intensity of the given input image channel relative to a summation of the input intensities of the N-channels of the input image.

15. An image processing method, comprising:
obtaining an N-channel input image;
determining fusion weights and fallback weights for each of the N-channels of the input image;
blending the fusion and fallback weights based on an amount of gradient information in the input image to generate blended weights;
modulating the blended weights for a plurality of frequency band representations of the input image;
applying the modulated blended weights to the corresponding frequency band representations of the input image to generate a plurality of output image frequency band representations; and
producing an output luma image, based on the plurality of output image frequency band representations.

16. The method of claim 15 , wherein the N-channel input image comprises an RGB-IR image.

17. The method of claim 15 , further comprising:
generating an output RGB image based on the output luma image.

18. The method of claim 17 , wherein generating an output RGB image based on the output luma image further comprises:
determining original color differences for pixels in the input image;
modulating the determined original color differences for pixels in the input image; and
adding the modulated determined original color differences for the pixels in the input image to the corresponding pixels in the output luma image to generate the output RGB image.

19. The method of claim 15 , wherein the frequency band representations of the input image are created by a difference of Gaussians (DoG) pyramid operation.

20. The method of claim 15 , wherein the method is performed, at least in part, by a Field Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (ASIC).

Assignees

• Apple Inc

Inventors

• Hayes Alex
• Romanenko Ilya

Classifications

• G06T7/10 Primary

  Segmentation; Edge detection (motion-based segmentation G06T7/215) · CPC title

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/10024

  Color image · CPC title