Method and device for generating neuron network compensated for loss due to pruning

Patent metadata

Field	Value
Publication number	US-2023124054-A1
Application number	US-202217964452-A
Country	US
Kind code	A1
Filing date	Oct 12, 2022
Priority date	Oct 19, 2021
Publication date	Apr 20, 2023
Grant date	—

Abstract

Embodiments relate to a method and device for generating a neural network that compensates for information loss due to pruning, including obtaining a trained neural network; pruning at least one neuron in the trained neural network; and updating one or more parameter values of a next layer in a pruned neural network based on one or more parameter values of at least one neuron among remaining neurons in a pruned target layer having the pruned neuron.

First claim

What is claimed is:

1 . A method for generating a neural network that compensates for information loss due to pruning performed by a processor, the method comprising:
obtaining a trained neural network; pruning at least one neuron in the trained neural network; and updating one or more parameter values of a next layer in a pruned neural network based on one or more parameter values of at least one neuron among remaining neurons in a pruned target layer having the pruned neuron.

2 . The method according to claim 1 , wherein the step of updating includes the steps of:
selecting another neuron or a combination of other neurons most similar to the pruned neuron among the remaining neurons in the target layer; computing a merging value based on one or more parameter values of the selected other neuron or one or more parameter values of a plurality of other neurons underlying the combination and one or more parameter values of the pruned neuron; and computing a merging matrix including the merging value based on one or more parameter values of remaining neurons in the pruned target layer and one or more parameter values of a neuron in an unpruned target layer.

3 . The method according to claim 2 , wherein the step of updating further includes the step of updating one or more parameter values of a neuron in the next layer in the pruned neural network based on one or more parameter values of the merging matrix and one or more parameter values of a neuron in the next layer in an unpruned neural network.

4 . The method according to claim 2 , wherein the other neuron most similar to the pruned neuron is a neuron having a most similar orientation among the remaining neurons in the pruned target layer,
the merging value is a ratio between the parameter value of the selected neuron and the parameter value of the pruned neuron.

5 . The method according to claim 2 , wherein the combination of other neurons most similar to the pruned neuron is one in which a result of combining two or more of the remaining neurons in the pruned target layer has the most similar orientation to the pruned neuron, and the combination is a sum of the one or more parameter values.

6 . The method according to claim 5 , wherein the merging matrix includes a plurality of merging values, and each of the plurality of merging values is a coefficient of a sum of one or more parameter values for respective sub neurons of the selected combination.

7 . The method according to claim 2 , wherein the merging matrix is computed by decomposing a matrix of one or more parameter values of neurons in the unpruned target layer into a matrix of one or more parameter values of the remaining neurons in the pruned target layer and the merging matrix through a matrix decomposition way.

8 . The method according to claim 2 , wherein one or more parameter values of neuron in the next layer in the pruned neural network is updated through the following equation,
W i+1 ′=Z i W i+1   [Equation]
wherein W i+1 ′ is a matrix of the parameter value of neuron in the next layer in the updated, pruned neural network, Z i is the merging matrix, and W i+1 is a matrix of one or more parameter values of neuron in the next layer in the un-updated, pruned neural network.

9 . The method according to claim 1 , wherein the neural network includes at least some of a plurality of fully connected layers and a plurality of convolutional layers,
the fully connected layer includes a node as the neuron, and the convolutional layer includes a filter as the neuron, the parameter of the neuron includes at least one of a node parameter and a filter parameter.

10 . The method according to claim 2 , wherein the neural network includes an activation function between the target layer and the next layer.

11 . The method according to claim 10 , wherein when the activation function is ReLU, the step of updating is performed when the merging matrix satisfies a preset specific condition,
the specific condition includes that the merging matrix Z i has only non-negative component value.

12 . The method according to claim 11 , wherein the specific condition further includes that the merging matrix has at most one positive component value per column.

13 . A non-transitory computer-readable recording medium which records a program for executing the method according to claim 1 .

Assignees

• Korea Inst Sci & Tech

Inventors

• Kim Suhyun

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/0464 Primary

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title