Electronic device for colorizing black and white image using GAN based model comprising transformer block and method for operation thereof

Patent metadata

Field	Value
Publication number	US-12524930-B2
Application number	US-202418422534-A
Country	US
Kind code	B2
Filing date	Jan 25, 2024
Priority date	Feb 3, 2023
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

In accordance with various embodiments, an electronic device for colorizing a black and white image using a Generative Adversarial Network (GAN)-based model comprising a transformer block includes a processor, wherein the processor is set to: obtain a black and white image including only first information about a luminance channel; and generate a pseudo color image including only second information about a chrominance channel by applying the black and white image to the GAN-based model, the GAN-based model includes a generator network including a plurality of transformer blocks for color conversion, a plurality of convolution layers, and a plurality of transpose convolution layers, the plurality of transformer blocks each include a Depth Wise Convolution (DWC) layer, a first Layer Normalization (LN) layer, a Window-based Multi-head Self Attention (W-MSA) layer, a second LN layer, and a Colorization Feed Forward (CFF) block. Other various embodiments are possible.

First claim

What is claimed is:

1 . An electronic device for colorizing a black and white image using a Generative Adversarial Network (GAN)-based model comprising a transformer block, the electronic device comprising a processor,
wherein the processor is set to: obtain a black and white image including only first information about a luminance channel; and generate a pseudo color image including only second information about a chrominance channel by applying the black and white image to the GAN-based model, the GAN-based model includes a generator network including a plurality of transformer blocks for color conversion, a plurality of convolution layers, and a plurality of transpose convolution layers, the plurality of transformer blocks each include a Depth Wise Convolution (DWC) layer, a first Layer Normalization (LN) layer, a Window-based Multi-head Self Attention (W-MSA) layer, a second LN layer, and a Colorization Feed Forward (CFF) block, the W-MSA layer includes a first-group MSA and a second-group MSA, and the first-group MSA is provided with a feature map divided into first-type windows and the second-group MSA is provided with a feature map divided into second-type windows obtained by shifting the first-type windows.

2 . The electronic device of claim 1 , wherein the first-group MSA and the second-group MSA are each composed of four heads.

3 . The electronic device of claim 2 , wherein the W-MSA layer transmits a value concatenating a result value of the first-group MSA and a result value of the second-group MSA to a next layer.

4 . The electronic device of claim 3 , wherein the generator network has an encoder-decoder-based architecture that uses the plurality of transformer blocks.

5 . The electronic device of claim 4 , wherein the GAN-based model is trained using a total loss L total considering all of a pixel wise (L1) loss function L L1 , a VGG loss function L VGG , and a WGAN loss function L wgan , and
the pixel wise (L1) loss function L L1 , the VGG loss function L VGG , the WGAN loss function L wgan , and the total loss L total are calculated from the following [Equation 1],

[

Equation
⁢

1

]



L
wgan

⁢

{

L
G

=

-

E

y
~

[

D
⁡
(

y
~

,
x

)

]

L
D

=

E
y

[

D
⁡
(

y
,
x

)

]

-

E

y
~

[

D
⁡
(

y
~

,
x

)

]

+

λ
×
GP

1

L

L
⁢
1

=



J
-

G
⁡
(
I
)



1

2

L

Assignees

• Univ Chosun Iacf

Inventors

• Lee Bum Shik
• Shafiq Muhammad Hamza

Classifications

• G06T11/10 Primary

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title