Image restoring circuit using neural network and image restoring method using the circuit

Patent metadata

Field	Value
Publication number	US-12524834-B2
Application number	US-202318530079-A
Country	US
Kind code	B2
Filing date	Dec 5, 2023
Priority date	Oct 16, 2023
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

An image restoring circuit includes a first restoring circuit including a first encoder and a first decoder, and configured to generate a first output image and first tensor data by restoring an input image; a second restoring circuit including a second encoder and a second decoder, and configured to restore the input image by using an output of the first encoder, an output of the first decoder, and the first tensor data to thereby generate a second output image; and a coupling circuit configured to generate second tensor data based on the output of the first encoder and the output of the first decoder and provide the second tensor data to the second encoder.

First claim

What is claimed is:

1 . An image restoring circuit comprising:
a first restoring circuit including a first encoder and a first decoder, and configured to generate a first output image and first tensor data by restoring an input image; a second restoring circuit including a second encoder and a second decoder, and configured to restore the input image by using an output of the first encoder, an output of the first decoder, and the first tensor data to thereby generate a second output image; and a coupling circuit configured to generate second tensor data based on the output of the first encoder and the output of the first decoder and provide the second tensor data to the second encoder.

2 . The image restoring circuit of claim 1 , wherein the first restoring circuit further comprises:
a first input circuit configured to extract first feature data from the input image and provide the first feature data to the first encoder; and a first output circuit configured to generate the first output image and the first tensor data based on the output of the first decoder and the input image.

3 . The image restoring circuit of claim 2 , wherein the second restoring circuit further comprises:
a second input circuit configured to extract second feature data from the input image and the first tensor data and provide the second feature data to the second encoder; and a second output circuit configured to generate the second output image based on an output of the second decoder and the input image.

4 . The image restoring circuit of claim 3 , wherein the second input circuit includes:
a third input circuit configured to extract the first feature data from the input image; a concatenation circuit configured to concatenate the first feature data and the first tensor data; and a fourth input circuit configured to process an output of the concatenation circuit and provide its output to the second encoder.

5 . The image restoring circuit of claim 3 , wherein the second output circuit includes:
a third output circuit configured to process the output of the second decoder; and an addition circuit configured to generate the second output image by adding an output of the third output circuit and the input image.

6 . The image restoring circuit of claim 1 , wherein the first encoder includes (N+1) first sub-encoders that are connected sequentially,
wherein the second decoder includes N first sub-decoders that are connected sequentially, wherein the second encoder includes (N+1) second sub-encoders that are connected sequentially, wherein the second decoder includes N second sub-decoders that are connected sequentially, and wherein N is a natural number.

7 . The image restoring circuit of claim 6 , wherein the coupling circuit includes N sub-coupling circuits,
wherein an i-th sub-coupling circuit processes an output of an i-th first sub-encoder and an output of an i-th first sub-decoder and provides processing results thereof to an i-th second sub-encoder, and wherein i is a natural number that is equal to or smaller than N.

8 . The image restoring circuit of claim 1 , wherein each of the first encoder and the second encoder includes an operation circuit that performs a deformable convolution operation.

9 . The image restoring circuit of claim 1 , further comprising:
a color correction circuit configured to generate scalar correction values for respective channels of the input image; and a multiplication circuit configured to generate a restored image by multiplying the scalar correction values with respective channels of the second output image.

10 . The image restoring circuit of claim 9 , wherein the color correction circuit includes:
a feature extraction circuit configured to extract features from the input image; a pooling circuit configured to perform a global average pooling operation on the features output from the feature extraction circuit; and a correction value generation circuit configured to generate the scalar correction values for the respective channels of the input image using an output of the pooling circuit.

11 . An image restoring method comprising:
performing a first restoration operation to generate a first output image and first tensor data by applying an input image to a first encoder and a first decoder; performing a coupling operation on an output of the first encoder and an output of the first decoder, thereby generating second tensor data; and performing a second restoration operation to generate the second output image by applying data, which is generated based on the input image and the first tensor data, and the second tensor data to a second encoder and a second decoder.

12 . The image restoring method of claim 11 , wherein performing the first restoration operation includes:
performing a first input operation to extract first feature data from the input image and provide the first feature data to the first encoder; and performing a first output operation to generate the first output image and the first tensor data by processing the output of the first decoder and the input image.

13 . The image restoring method of claim 11 , wherein performing the second restoration operation includes:
performing a second input operation to extract second feature data from the input image and the first tensor data and provide the second feature data to the second encoder; and performing a second output operation to generate the second output image based on an output of the second decoder and the input image.

14 . The image restoring method of claim 13 , wherein performing the second input operation includes:
performing a third input operation to extract first feature data from the input image; performing a concatenation operation to concatenate the first feature data and the first tensor data; and performing a fourth input operation to generate the second feature data using an output of the concatenation operation and provide the second feature data to the second encoder.

15 . The image restoring method of claim 13 , wherein performing the second output operation includes:
performing a third output operation to process the output of the second decoder; and performing an addition operation to generate the second output image by adding an output of the third output operation and the input image.

16 . The image restoring method of claim 11 , wherein performing the first restoration operation includes performing a deformable convolution operation in the first encoder, or performing the second restoration operation includes performing a deformable convolution operation in the second encoder.

17 . The image restoring method of claim 11 , further comprising:
performing a color correction operation to generate scalar correction values for respective channels of the input image; and multiplying the scalar correction values with the respective channels of the input image to generate a restored image.

18 . The image restoring method of claim 17 , wherein performing the color correction operation includes:
extracting feature data from the input image; performing global average pooling operations on respective channels of the input image; and generating the scalar correction values for the respective channels from outputs of the global average pooling operations.

19 . The image restoring method of claim 11 , wherein the first restoring operation, the coupling operation, and the second restoring operation are performed by using a neural network circuit, and t

Assignees

• Sk Hynix Inc
• Seoul Nat Univ R&Db Foundation

Inventors

• Oh Youngjin
• Park Guyong
• Chung Haesoo
• Cho Nam Ik

Classifications

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T9/002

  using neural networks · CPC title

• G06T2207/20056

  Discrete and fast Fourier transform, [DFT, FFT] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06T2207/10024

  Color image · CPC title