Neural processing device and method for pruning thereof

Patent metadata

Field	Value
Publication number	US-2022300816-A1
Application number	US-202217655348-A
Country	US
Kind code	A1
Filing date	Mar 17, 2022
Priority date	Mar 19, 2021
Publication date	Sep 22, 2022
Grant date	—

Abstract

A neural processing device and method for pruning thereof are provided. The neural processing device includes a processing unit configured to perform calculations, an L0 memory configured to store input and output data of the processing unit, wherein the input and output data include a two-dimensional weight matrix and a weight manipulator configured to receive the two-dimensional weight matrix and partition it into preset sizes to thereby generate partitioned matrices, to generate a pruning matrix by pruning the partitioned matrix, and to transmit the pruning matrix to the processing unit.

First claim

What is claimed is:

1 . A neural processing device comprising:
a processing unit configured to perform calculations; an L0 memory configured to store input and output data of the processing unit, wherein the input and output data include a two-dimensional weight matrix; and a weight manipulator configured to receive the two-dimensional weight matrix and partition it into preset sizes to thereby generate partitioned matrices, to generate a pruning matrix by pruning the partitioned matrix, and to transmit the pruning matrix to the processing unit.

2 . The neural processing device of claim 1 , wherein the processing unit comprises a PE array that performs two-dimensional calculations and that includes at least one processing element, and
the processing element receives an input in SIMD (Single Instruction/Multiple Data).

3 . The neural processing device of claim 2 , wherein the size of the partitioned matrix is determined based on the width of the SIMD of the processing element.

4 . The neural processing device of claim 2 , wherein the weight manipulator comprises:
a width identifier configured to check the width of the SIMD of the processing element and generate a confirmation signal; a weight initializer configured to receive the confirmation signal and initialize the two-dimensional weight matrix; a matrix divider configured to partition the two-dimensional weight matrix according to the width of the SIMD of the processing element and generate the partitioned matrices; and a pruner configured to generate the pruning matrix by pruning the partitioned matrix.

5 . The neural processing device of claim 4 , wherein the partitioned matrix contains at least one group, and
the number of elements in the group is equal to the width of the SIMD of the processing element.

6 . The neural processing device of claim 5 , wherein the pruner is configured to:
generate a representative value of the group, and compare the representative value with a threshold value and determine whether to convert the group into a zero group.

7 . The neural processing device of claim 6 , wherein the two-dimensional weight matrix contains weights as elements, and
the weights and the threshold value are trained via an artificial neural network.

8 . The neural processing device of claim 6 , wherein the threshold value comprises an initial threshold value and an updated threshold value, and
the pruner generates dry run information by performing pruning through the initial threshold value, and generates the pruning matrix through the updated threshold value.

9 . The neural processing device of claim 8 , wherein the weight manipulator further comprises a load balance unit configured to receive the dry run information and generate the updated threshold value.

10 . The neural processing device of claim 9 , wherein the load balance unit comprises:
a zero group counter configured to receive the dry run information, to count zero groups, and to generate counting information; a threshold updater configured to receive the counting information and generate the updated threshold value.

11 . The neural processing device of claim 8 , wherein the updated threshold value comprises a partitioned matrix threshold value corresponding to each partitioned matrix.

12 . A method for pruning of a neural processing device, comprising:
checking the width of SIMD of a processing element, initializing a two-dimensional weight matrix, partitioning the two-dimensional weight matrix into partitioned matrices based on the width of the SIMD, and pruning the partitioned matrix using a threshold value.

13 . The method for pruning of a neural processing device of claim 12 , wherein the partitioned matrix contains at least one group, and the group has a size equal to the width of the SIMD, and
the pruning comprises: generating a representative value of the group, and changing the group to a zero group if the representative value is less than or equal to the threshold value.

14 . The method for pruning of a neural processing device of claim 13 , wherein the representative value comprises any one of a mean value, a minimum value, a maximum value, a median value, and a root mean square (RMS) value.

15 . The method for pruning of a neural processing device of claim 13 , wherein the two-dimensional weight matrix contains at least one weight,
the weight and the threshold value are trained via an artificial neural network, and the threshold value is trained in a direction in which the number of zero groups is uniformly distributed for each partitioned matrix.

16 . The method for pruning of a neural processing device of claim 13 , wherein the threshold value comprises an initial threshold value and an updated threshold value, and
the pruning comprises: generating dry run information by pruning with the initial threshold value, generating counting information by counting the zero groups based on the dry run information, generating the updated threshold value based on the counting information, and generating the pruning matrix by pruning based on the updated threshold value.

17 . The method for pruning of a neural processing device of claim 16 , wherein the generating the updated threshold value comprises:
selecting a partitioned matrix having the largest number of zero groups in the counting information as a reference partitioned matrix, maintaining an initial threshold value of the reference partitioned matrix, and adjusting threshold values of the remaining partitioned matrices other than the reference partitioned matrix.

18 . The method for pruning of a neural processing device of claim 12 , wherein the pruning comprises generating a pruning matrix by pruning the partitioned matrix, and
the pruning method of a neural processing device, further comprising: performing, by the processing element, calculations using the pruning matrix.

19 . The method for pruning of a neural processing device of claim 18 , wherein the pruning matrix contains at least one group,
the at least one group comprises a zero group in which all elements are zero and a non-zero group in which at least one non-zero element is included, and the performing calculations comprises skipping the calculation of the zero group.

20 . The method for pruning of a neural processing device of claim 12 , wherein the two-dimensional weight matrix is a form obtained by rearranging a four-dimensional tensor in two dimensions.

Assignees

• Rebellions Inc

Inventors

• Oh Jinwook

Classifications

• G06F9/3834 Primary

  Maintaining memory consistency · CPC title

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/063

  using electronic means · CPC title

• G06F9/30036

  Instructions to perform operations on packed data, e.g. vector, tile or matrix operations · CPC title

• G06F9/3887

  controlled by a single instruction for multiple data lanes [SIMD] · CPC title