Image fusion architecture

Patent metadata

Field	Value
Publication number	US-11798146-B2
Application number	US-202016987202-A
Country	US
Kind code	B2
Filing date	Aug 6, 2020
Priority date	Aug 6, 2020
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

Embodiments relate to circuitry for temporal processing and image fusion. An image fusion circuit receives captured images, and generates corresponding image pyramids. The generated image pyramids are raster or tiled processed, and stored in memory. A fusion module receives a first and second image pyramids from the memory, warps the first image pyramid based upon the second image pyramid, and fuses the warped first image pyramid with the second image pyramid to generate a fused image pyramid, which may be used for further processing, and may also be stored back into the memory. Because pyramid generation occurs prior to warping and fusion, and by allowing fused image pyramids to be stored back into memory, the image fusion circuitry is configurable to implement a variety of temporal processing functions involving different image fusion combinations.

First claim

What is claimed is:

1. An image signal processor, comprising:
a pyramid generation circuit configured to:
receive a plurality of images, and
generate a plurality of pyramids corresponding to the plurality of received images, each comprising a plurality of stages corresponding to sequentially downscaled versions of a respective image of the plurality of received images; and

a data routing circuit coupled to the pyramid generation circuit to receive the plurality of pyramids, and configured to:
receive one or more parameters specifying a configuration mode of a plurality of available configuration modes for the image signal processor, each corresponding to a different temporal image processing application, and
for each pyramid of the plurality of pyramids, route the pyramid to be stored into a memory where a time until the pyramid is to be used for image fusion in accordance with the specified configuration mode exceeds a length of time, or route the pyramid to a cache separate from the memory to bypass the memory where a time until the pyramid is to be used for image fusion in accordance with the specified configuration mode is less than the length of time; and

an image fusion processor coupled to the data routing circuit configured to:
receive a first pyramid and a second pyramid selected based upon the specified configuration mode, wherein at least one of the first pyramid or the second pyramid is one of the plurality of pyramids generated by the pyramid generation circuit;
warp the first pyramid according to one or more warping parameters to generate a warped first pyramid; and
fuse the warped first pyramid and the second pyramid to generate a fused image.

2. The image signal processor of claim 1 , wherein the image fusion processor is further configured to:
responsive to a first condition associated with the configuration mode being met, output the fused image to a noise reduction circuit; and
responsive to a second condition associated with the configuration mode being met, transmit an image pyramid of the fused image to the data routing circuit for storage into the memory.

3. The image signal processor of claim 2 , further comprising a post-processor circuit coupled to the noise reduction circuit, the post-processor circuit configured to perform post-processing of an unscaled single-color version of the fused image and a first downscaled multi-color version of the fused image to obtain a post-processed version of the fused image.

4. The image signal processor of claim 1 , wherein each pyramid of the plurality of pyramids further comprises a first stage corresponding to a single-color unscaled version of the respective image in addition to the plurality of stages corresponding to he sequentially downscaled versions of the respective image, wherein each of the sequentially downscaled versions of the respective image is a multi-colored version of the respective image.

5. The image signal processor of claim 1 , wherein the first pyramid was generated from a first image, and the second pyramid was generated from a second image, and further comprising an image registration circuit configured to:
receive the first image;
generate a set of keypoints indicating features in the first image; and
wherein the one or more warping parameters are derived from a model describing correspondence between the first image and the second image, and are determined by comparing the generated set of keypoints with another set of keypoints of the second image.

6. The image signal processor of claim 1 , wherein the fused image comprises an image pyramid having a first unscaled single-color version and a multi-color downscaled version of the fused image.

7. The image signal processor of claim 1 , wherein the data routing circuit comprises:
a multiplexor configured to receive the plurality of pyramids, and for each pyramid, select between formatting data of the pyramid in raster format or tile format based upon whether the pyramid is to be warped as part of an image fusion process in accordance with the configuration mode; and
a direct memory access (DMA) module configured to receive the data of the pyramid in raster format or tile format, and route the received data of the pyramid to be stored in the memory or to bypass the memory where at least a portion of the received data of the pyramid is stored in a cache separate from the memory.

8. The image signal processor of claim 1 , wherein at least one of the first pyramid or the second pyramid received by the image fusion processor corresponds to an image pyramid of a previous fused image.

9. The image signal processor of claim 1 , wherein the image fusion processor is configured to fuse the first and second pyramids by:
blending at least one pixel of an unscaled single-color version of the first image corresponding to a first stage of the first pyramid with a corresponding pixel of an unscaled single-color version of the second image corresponding to a first stage of the second pyramid to generate an unscaled single-color version of the fused image; and
blending at least one pixel of at least one downscaled version of the first image of the first pyramid with a corresponding pixel of a corresponding downscaled version of the second image of the second pyramid to generate a first downscaled multi-color version of the fused image.

10. The image signal processor of claim 9 , wherein:
each pixel of the unscaled single-color version of the first image is associated with a first confidence value, and each pixel of the unscaled single-color version of the second image is associated with a second confidence value, and
wherein the image fusion processor is configured to blend the at least one pixel of the unscaled single-color version of the first image with the corresponding pixel of the unscaled single-color version of the second image based upon the first and second confidence values.

11. The image signal processor of claim 1 , wherein the first pyramid is formatted in tile format, and the second pyramid is formatted in raster format.

12. The image signal processor of claim 1 , wherein the pyramid is routed by the data routing circuit based upon an order in the plurality of pyramids in which the pyramid was received.

13. A method for image fusion, comprising:
receiving a plurality of images at a pyramid generation circuit, and
generating, at the pyramid generation circuit, a plurality of pyramids corresponding to the plurality of received images, each comprising a plurality of stages corresponding to sequentially downscaled versions of a respective image of the plurality of received images;
at a data routing circuit coupled to the pyramid generation circuit:
receiving one or more parameters specifying a configuration mode of a plurality of available configuration modes, each corresponding to a different temporal image processing application;
for each pyramid of the plurality of pyramids, route the pyramid to be stored into a memory where a time until the pyramid is to be used for image fusion in accordance with the specified configuration mode exceeds a length of time, or route the pyramid to a cache separate from the memory to bypass the memory where a time until the pyramid is to be used for image fusion in accordance with the specified configuration mode;

receiving, at an image fusion processor, a first pyramid and a second pyramid selected based upon the specified configuration mode, wherein at least one of the first pyramid or the second pyramid is one of the plurality of pyramids generated by the pyramid generation circuit;
warping the first pyramid according to one or more warping parameters to gen

Assignees

• Apple Inc

Inventors

• Smirnov Maxim
• Pope David R

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T3/0093

  Physics · mapped topic

• G06T5/002

  Physics · mapped topic

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/10024

  Color image · CPC title