Systems and methods of spatiotemporal image noise reduction for multispectral image data

Patent metadata

Field	Value
Publication number	US-11265490-B2
Application number	US-202016901490-A
Country	US
Kind code	B2
Filing date	Jun 15, 2020
Priority date	Jun 13, 2019
Publication date	Mar 1, 2022
Grant date	Mar 1, 2022

Abstract

Devices, methods, and non-transitory program storage devices for spatiotemporal image noise reduction are disclosed, comprising: maintaining an accumulated image in memory; and obtaining a first plurality of multispectral images (e.g., RGB-IR images). For each image in the first plurality of multispectral images, the method may: calculate a multispectral guide image for the current image; calculate blending weights for the current image; apply the calculated blending weights to each channel of the current image to generate a denoised current image; and update the accumulated image based on pixel differences between the denoised current image and the accumulated image. In some embodiments, additional images (e.g., the accumulated image and/or other images captured prior to or after a given current image) may also be included in the denoising operations for a given current image. Finally, the method may generate a denoised output image for each input image, based on the updated accumulated image.

First claim

What is claimed is:

1. A device, comprising:
a memory;
one or more image capture devices;
a user interface; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
maintain an accumulated image in the memory;
obtain a first plurality of multispectral images; and
for each image in the first plurality of multispectral images, perform the following operations:
calculate a multispectral guide image for a current image of the first plurality of multispectral images;
calculate a multispectral guide image for the accumulated image;
determine first differences between pixels within first blocks of pixels in the multispectral guide image for the current image and second blocks of pixels in the multispectral guide image for the current image;
determine second differences between pixels within blocks of pixels in the multispectral guide image for the accumulated image and blocks of pixels in the multispectral guide image for the current image;
calculate blending weights for the current image and the accumulated image based, at least in part, on the determined first differences and second differences;
denoise the current image based on the calculated blending weights;
update the accumulated image based on the denoised current image; and
generate a noise-reduced output image based on the updated accumulated image.

2. The device of claim 1 , wherein the first plurality of multispectral images comprises RGB-IR images.

3. The device of claim 2 , wherein the instructions to determine the first differences and the second differences are executed prior to any decorrelation of RGB image data from IR image data.

4. The device of claim 1 , wherein the blocks of pixels comprise neighborhoods of pixels around a central pixel, and wherein the blocks of pixels have dimensions that are determined based, at least in part, on an amount of available pixel lines in memory during execution.

5. The device of claim 1 , wherein the multispectral guide image for the accumulated image is formed by: performing a maximum operation across the image channels of the accumulated image at each pixel location.

6. The device of claim 1 , wherein at least one of the first differences or the second differences comprise: noise-normalized differences based on an expected noise level.

7. The device of claim 1 , wherein the instructions further comprise instructions causing the one or more processors, for a given current image of the first plurality of multispectral images, to:
calculate a multispectral guide image for a third image;
determine third differences between pixels within blocks of pixels in the multispectral guide image for the third image and blocks of pixels in the multispectral guide image for the given current image; and
calculate blending weights for the third image based, at least in part, on the determined third differences.

8. The device of claim 7 , wherein the third image comprises an image captured after the given current image.

9. The device of claim 1 , wherein the blending weights for the current image and the accumulated multispectral guide image are normalized prior to execution of the instructions to denoise the current image based on the calculated blending weights.

10. A non-transitory computer readable medium comprising computer readable instructions configured to cause one or more processors to:
maintain an accumulated image in a memory;
obtain a first plurality of N-channel multispectral images, wherein the first plurality of N-channel multispectral images comprise images having at least one visible light channel and one non-visible light channel; and
for each image in the first plurality of N-channel multispectral images, perform the following operations:
calculate a multispectral guide image for a current image of the first plurality of N-channel multispectral images based, at least in part, on the non-visible light channel from the current image;
calculate blending weights for the current image based, at least in part, on the multispectral guide image for the current image;
apply the calculated blending weights to each of the N-channels of the current image to generate a denoised current image;
update the accumulated image based, at least in part, on pixel differences between the denoised current image and the accumulated image; and
generate a noise-reduced output image based on the updated accumulated image.

11. The non-transitory computer readable medium of claim 10 , wherein the first plurality of multispectral images comprises RGB-IR images.

12. The non-transitory computer readable medium of claim 11 , wherein the instructions to apply the calculated blending weights to the current image to generate a denoised current image further comprise instructions to apply the calculated blending weights to each of N-channels of the current image in parallel.

13. The non-transitory computer readable medium of claim 10 , wherein the multispectral guide image for a given current image is formed by: performing a maximum operation across the N image channels of the given current image.

14. The non-transitory computer readable medium of claim 10 , wherein the instructions to calculate blending weights for the current image based, at least in part, on the multispectral guide image further comprise instructions to: determine differences between pixels within first blocks of pixels in the multispectral guide image for the current image and second blocks of pixels in the multispectral guide image for the current image.

15. The non-transitory computer readable medium of claim 10 , wherein the instructions to update the accumulated image based, at least in part, on pixel differences between the denoised current image and the accumulated image further comprise instructions to:
update the accumulated image using a weighting parameter determined based on a noise-normalized difference between the denoised current image and the accumulated image.

16. An image processing method, comprising:
maintaining an accumulated image in a memory;
obtaining a first plurality of N-channel multispectral images, wherein the first plurality of N-channel multispectral images comprise images having at least one visible light channel and one non-visible light channel; and
for each image in the first plurality of N-channel multispectral images, perform the following operations:
calculating a multispectral guide image for a current image of the first plurality of N-channel multispectral images based, at least in part, on the non-visible light channel from the current image;
calculating blending weights for the current image based, at least in part, on the multispectral guide image for the current image;
applying the calculated blending weights to each of the N-channels of the current image to generate a denoised current image;
updating the accumulated image based, at least in part, on pixel differences between the denoised current image and the accumulated image; and
generating a noise-reduced output image based on the updated accumulated image.

17. The method of claim 16 , wherein applying the calculated blending weights to the current image to generate a denoised current image further comprises applying the calculated blending weights to each of N-channels of the current image in parallel.

18. The method of claim 16 , wherein the multispectral guide image is formed by: performing a maximum operation across the N image channels of the current image.

Assignees

• Apple Inc

Inventors

• Romanenko Ilya
• Hayes Alex

Classifications

• H04N5/21 Primary

  Circuitry for suppressing or minimising disturbance, e.g. moiré or halo · CPC title

• G06F2218/04

  Denoising · CPC title

• H04N25/60

  Noise processing, e.g. detecting, correcting, reducing or removing noise · CPC title

• H04N5/33

  Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20) · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title