Asymmetric multi-modal image fusion

Patent metadata

Field	Value
Publication number	US-2025182246-A1
Application number	US-202318524531-A
Country	US
Kind code	A1
Filing date	Nov 30, 2023
Priority date	Nov 30, 2023
Publication date	Jun 5, 2025
Grant date	—

Abstract

A system for performing asymmetric multi-modal image fusion is configurable to (i) access a first image associated with a first imaging modality; (ii) decompose the first image into a first base layer and a first detail layer; (iii) determine a weight map based on pixel signals of the first image; and (iv) generate an output image by performing image fusion using the first base layer, the first detail layer, and a second detail layer associated with a second imaging modality that is different from the first imaging modality, where the weight map modifies the first detail layer and the second detail layer in the image fusion.

First claim

We claim:

1 . A system for performing asymmetric multi-modal image fusion, the system comprising:
one or more processors; and one or more computer-readable recording media that store instructions that are executable by the one or more processors to configure the system to:
access a first image associated with a first imaging modality;
decompose the first image into a first base layer and a first detail layer;
determine a weight map based on pixel signals of the first image; and
generate an output image by performing image fusion using the first base layer, the first detail layer, and a second detail layer associated with a second imaging modality that is different from the first imaging modality, wherein the weight map modifies the first detail layer and the second detail layer in the image fusion.

2 . The system of claim 1 , wherein the first imaging modality comprises a visible light imaging modality.

3 . The system of claim 1 , wherein the second imaging modality comprises a thermal imaging modality.

4 . The system of claim 1 , wherein the first base layer comprises a first low-frequency component of the first image, and wherein the first detail layer comprises a first high-frequency component of the first image.

5 . The system of claim 4 , wherein the first low-frequency component is determined by:
generating a downscaled first image by downscaling the first image; generating a filtered downscaled first image by applying a blurring or smoothing filter to the downscaled first image; and generating the first low-frequency component by upscaling the filtered downscaled first image.

6 . The system of claim 4 , wherein the first high-frequency component is determined by subtracting the first low-frequency component from the first image.

7 . The system of claim 4 , wherein the second detail layer comprises a second high-frequency component of a second image associated with the second imaging modality.

8 . The system of claim 7 , wherein the second detail layer is obtained by:
generating a downscaled second image by downscaling the second image; generating a filtered downscaled second image by applying a blurring or smoothing filter to the downscaled second image; generating a second low-frequency component by upscaling the filtered downscaled second image; and subtracting the second low-frequency component from the second image.

9 . The system of claim 8 , wherein the image fusion refrains from using the second low-frequency component.

10 . The system of claim 7 , wherein the first image and the second image comprise geometry-corrected images.

11 . The system of claim 1 , wherein the weight map comprises an alpha map.

12 . The system of claim 11 , wherein values of the alpha map are determined by evaluating a negative exponential function at each of the pixel signals of the first image.

13 . The system of claim 1 , wherein the image fusion further uses a pedestal component to shift pixel values into a positive domain.

14 . The system of claim 13 , wherein the pedestal component is based on differences between a maximum pedestal value and pixel values of the first base layer.

15 . The system of claim 13 , wherein the pedestal component is modified by the weight map.

16 . A system for performing asymmetric multi-modal image fusion, the system comprising:
one or more processors; and one or more computer-readable recording media that store instructions that are executable by the one or more processors to configure the system to:
access a first image associated with a first imaging modality;
decompose the first image into a first low-frequency component and a first high-frequency component;
access a second image associated with a second imaging modality that is different from the first imaging modality;
decompose the second image into a second low-frequency component and a second high-frequency component;
generate a fused detail layer by fusing the first high-frequency component with the second high-frequency component using a weight map based on pixel signals of the first image; and
generate an output image by combining the fused detail layer with the first low-frequency component.

17 . The system of claim 16 , wherein generating the output image refrains from using the second low-frequency component.

18 . The system of claim 16 , wherein values of the weight map are determined by evaluating a negative exponential function at each of the pixel signals of the first image.

19 . The system of claim 16 , wherein the output image is generated by applying a pedestal component to shift pixel values into a positive domain.

20 . A system for performing asymmetric multi-modal image fusion, the system comprising:
one or more processors; and one or more computer-readable recording media that store instructions that are executable by the one or more processors to configure the system to:
access a first image associated with a first imaging modality;
determine a per-pixel signal quality measurement of the first image; and
generate an output image at least by combining a component of the first image with a component of a second image associated with a second imaging modality that is different from the first imaging modality, wherein a per-pixel contribution of the component of the second image is negatively related to the per-pixel signal quality measurement of the first image.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Maekelae Christian Markus
• Edmonds Christopher Douglas

Classifications

• G06T2207/10048

  Infrared image · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T5/20

  using local operators · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title