Deep neural network slimming device and operating method thereof

Patent metadata

Field	Value
Publication number	US-2023334321-A1
Application number	US-202217976655-A
Country	US
Kind code	A1
Filing date	Oct 28, 2022
Priority date	Apr 14, 2022
Publication date	Oct 19, 2023
Grant date	—

Abstract

Disclosed are a deep neural network lightweight device based on batch normalization, and a method thereof. The deep neural network lightweight device based on batch normalization includes a memory that stores at least one data and at least one processor that executes a network lightweight module. When executing the network lightweight module, the processor performs learning on an input neural network based on sparsity regularization to adaptively determine at least one parameter of the sparsity regularization, performs pruning on the learning result, and performs fine tuning on the pruning result.

First claim

What is claimed is:

1 . A deep neural network lightweight device based on batch normalization, the device comprising:
a memory configured to store at least one data; and at least one processor configured to execute a network lightweight module, wherein, when executing the network lightweight module, the processor is configured to: perform learning on an input neural network based on sparsity regularization to adaptively determine at least one parameter of the sparsity regularization; perform pruning on the learning result; and perform fine tuning on the pruning result.

2 . The device of claim 1 , wherein the processor is configured to:
calculate a task loss and a regularization loss; perform backpropagation based on the calculation result; and perform the learning based on the backpropagation result.

3 . The device of claim 2 , wherein the sparsity regularization is transformed L1 (TL1) regularization, and
wherein the TL1 regularization is expressed as

P
a

(
x
)

=

∑

i
=
1

n

(

a
+
1

)

⁢

❘
"\[LeftBracketingBar]"

x
i

❘
"\[RightBracketingBar]"

a
+

❘
"\[LeftBracketingBar]"

x
i

❘
"\[RightBracketingBar]"

.

4 . The device of claim 3 , wherein the task loss is expressed as Σ x,y I(f(x,W),y), and
wherein the regularization loss is expressed as λΣ γ g(γ).

5 . The device of claim 4 , wherein the processor performs the learning by adaptively determining a parameter ‘a’.

6 . The device of claim 5 , wherein the processor determines the parameter ‘a’ based on a learning batch ‘x’, a scaling factor ‘γ’ of the batch normalization, and a target pruning ratio ‘p’.

7 . A deep neural network lightweight method based on batch normalization, the method comprising:
performing learning on an input neural network based on sparsity regularization; performing pruning on the learning result; and performing fine tuning on the pruning result, wherein the performing of the learning based on the sparsity regularization includes: adaptively determining at least one parameter of the sparsity regularization.

8 . The method of claim 7 , wherein the performing of the learning based on the sparsity regularization includes:
calculating a task loss and a regularization loss; and performing backpropagation after calculating a total loss from the calculated task loss and the calculated regularization loss.

9 . The method of claim 8 , wherein the sparsity regularization is transformed L1 (TL1) regularization, and
wherein the TL1 regularization is expressed as

P
a

(
x
)

=

∑

i
=
1

n

(

a
+
1

)

⁢

❘
"\[LeftBracketingBar]"

x
i

❘
"\[RightBracketingBar]"

a
+

❘
"\[LeftBracketingBar]"

x
i

❘
"\[RightBracketingBar]"

.

10 . The method of claim 9 , wherein the task loss is expressed as Σ x,y I(f(x,W),y).

11 . The method of claim 10 , wherein the regularization loss is expressed as λΣ γ g(γ).

12 . The method of claim 11 , wherein the adaptively determining of the at least one parameter of the sparsity regularization includes:
adaptively determining a parameter ‘a’.

13 . The method of claim 12 , wherein the adaptively determining of the parameter ‘a’ includes:
receiving a learning batch ‘x’, a scaling factor ‘γ’ of the batch normalization, and a target pruning ratio ‘p’;
sorting the scaling factor ‘γ’;
assigning a parameter ‘th’ by calculating a value corresponding to the target pruning ratio ‘p’ in the sorted scaling factor ‘γ’; and
calculating the parameter ‘a’ from the assigned parameter ‘th’.

14 . The method of claim 13 , wherein the calculating of the parameter ‘a’ from the assigned parameter ‘th’ satisfies a condition of

∂

P
a

(
x
)

∂
x

⌋

x
=
th

Assignees

• Electronics & Telecommunications Res Inst

Inventors

• Lee Su Woong
• Ko Jong-Gook
• Yoo Wonyoung
• Lee Seungjae
• Lee Yongsik
• Lee Juwon
• Jung Da-Un

Classifications

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title