Image processing device, electronic device having the same, and operating method thereof

Patent metadata

Field	Value
Publication number	US-12475535-B2
Application number	US-202318305625-A
Country	US
Kind code	B2
Filing date	Apr 24, 2023
Priority date	Sep 19, 2022
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

A method includes acquiring a current frame, a noise dispersion map for the current frame, and a first denoised frame for a previous frame; generating a weighted first denoised frame based on the noise dispersion map, the current frame, and the first denoised frame using a first neural network; generating an initial fused image based on the current frame and the weighted first denoised frame using a second neural network; and generating a second denoised frame for the current frame based on the initial fused image using a third neural network.

First claim

What is claimed is:

1 . A method comprising:
acquiring a current frame, a noise dispersion map for the current frame, and a first denoised frame for a previous frame; generating a weighted first denoised frame based on the noise dispersion map, the current frame, and the first denoised frame using a first neural network; generating an initial fused image based on the current frame and the weighted first denoised frame using a second neural network; and generating a second denoised frame for the current frame based on the initial fused image using a third neural network.

2 . The method of claim 1 , wherein at least one of the first neural network, the second neural network, and the third neural network includes a Convolutional Neural Network (CNN).

3 . The method of claim 1 , wherein the generating the weighted first denoised frame includes:
calculating an absolute value of a difference between the current frame and the first denoised frame; concatenating the absolute value and the noise dispersion map to obtain first concatenated data; calculating a weight map based on the first concatenated data using the first neural network; and multiplying the weight map by the first denoised frame to output the weighted first denoised frame.

4 . The method of claim 3 , wherein the noise dispersion map comprises one channel, the absolute value comprises three channels, and the first concatenated data comprises four channels.

5 . The method of claim 4 , wherein the generating the weighted first denoised frame includes applying the first neural network to the four channels of the first concatenated data.

6 . The method of claim 1 , wherein the first neural network includes a sigmoid activation function.

7 . The method of claim 1 , wherein the generating the initial fused image includes:
concatenating the current frame, the noise dispersion map, and the weighted first denoised frame to obtain second concatenated data; and generating the initial fused image based on the second concatenated data using the second neural network.

8 . The method of claim 7 , wherein the noise dispersion map comprises one channel, the current frame comprises three channels, the weighted first denoised frame comprises three channels, and the second concatenated data comprises seven channels.

9 . The method of claim 1 , wherein the generating the second denoised frame for the current frame includes:
concatenating the noise dispersion map, the initial fused image, and the first denoised frame to obtain third concatenated data; calculating a fusion weight map based on the third concatenated data using the third neural network; and fusing the initial fused image, the fusion weight, and the first denoised frame to generate the second denoised frame for the current frame.

10 . The method of claim 9 , wherein the noise dispersion map comprises one channel, the first denoised frame comprises three channels, the initial fused image comprises three channels, and the third concatenated data comprises seven channels.

11 . An image processing device comprising:
a processor configured to execute a denoising module; and a memory device configured to store the denoising module, wherein the denoising module includes: a first neural network for generating a weighted frame for a denoised frame for a previous frame; a second neural network for generating an initial fused image for the weighted frame using a current frame and the weighted frame; and a third neural network for fusing the current frame, the initial fused image, and the denoised frame for the previous frame.

12 . The image processing device of claim 11 , wherein the denoised frame for the previous frame comprises three channels, the current frame comprises three channels, a noise dispersion map for the current frame comprises one channel, and the weighted frame comprises three channels.

13 . The image processing device of claim 12 , wherein the current frame comprises three channels, the weighted frame comprises three channel, and the noise dispersion map comprises one channel, and the initial fused image comprises three channel.

14 . The image processing device of claim 13 , wherein the fused image comprises three channels, the denoised frame for the previous frame comprises three channels, and the noise dispersion map comprises one channel, and a denoised frame for the current frame comprises three channels.

15 . The image processing device of claim 11 , further comprising:
a first input buffer storing a noise dispersion map; a second input buffer storing the current frame; a third input buffer storing the denoised frame for the previous frame; and an output buffer storing a denoised frame for the current frame.

16 . An electronic device comprising:
a memory device configured to store a denoising module removing video/multi-frame noise; a memory controller configured to control the memory device; and a processor configured to execute the denoising module removing noise from an input image, wherein the denoising module performs a recurrent three-stage gradual temporal fusion using a convolutional neural network (CNN).

17 . The electronic device of claim 16 , wherein the denoising module includes a first neural network, a second neural network, and a third neural network.

18 . The electronic device of claim 17 , wherein the first neural network calculates an absolute value of a difference between first denoised frames for a previous frame and a current frame, concatenates the absolute value with a noise dispersion map for the current frame to obtain first concatenated data, calculates a weight map based on the first concatenated data, and multiplies the weight map by the first denoised frame to output a weighted first denoised frame.

19 . The electronic device of claim 18 , wherein the second neural network concatenates the current frame, the noise dispersion map, and the weighted first denoised frame to obtain second concatenated data, and outputs an initial fused image based on the second concatenated data.

20 . The electronic device of claim 19 , wherein the third neural network concatenates the noise dispersion map, the initial fused image, and the first denoised frame to obtain third concatenated data, calculates a fusion weight map based on the third concatenated data, and fuses the initial fused image, the fusion weight map, and the first denoised frame to output a second denoised frame for the current frame.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Guo Kai
• Choi Seungwon
• Choi Jongseong

Classifications

• G06T5/70 Primary

  Denoising; Smoothing · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title