Neural network operation method and apparatus with mapping orders

Patent metadata

Field	Value
Publication number	US-12314843-B2
Application number	US-202117369036-A
Country	US
Kind code	B2
Filing date	Jul 7, 2021
Priority date	Feb 25, 2021
Publication date	May 27, 2025
Grant date	May 27, 2025

Abstract

A neural network operation apparatus includes an input register to store an input feature map, a processing element array including a processing element to perform an operation based on the input feature map and a weight matrix, and a controller to map a portion of the input feature map and a portion of the weight matrix, both on which the operation is to be performed, to the processing element.

First claim

What is claimed is:

1. A neural network operation apparatus, comprising:
an input register configured to store an input feature map;
a processing element array comprising a processing element configured to perform an operation based on the input feature map and a weight matrix; and
a controller configured to map a portion of the input feature map and a portion of the weight matrix, both on which the operation is to be performed, to the processing element,
wherein a size of the processing element array is determined based on a height of an output feature map, a width of the output feature map, and a size of the weight matrix.

2. The neural network operation apparatus of claim 1 , wherein the processing element comprises:
a multiplexer configured to multiplex the portion of the input feature map; and
a shifter configured to shift an output of the multiplexer based on the portion of the weight matrix.

3. The neural network operation apparatus of claim 1 , wherein the processing element array comprises two-dimensional (2D) processing element planes, and
a size of each of the processing element planes is determined based on the height of the output feature map and the width of the output feature map.

4. The neural network operation apparatus of claim 1 , wherein the controller is configured to:
map a first weight included in the weight matrix and a first portion of the input feature map corresponding to the first weight to the processing element, and
map a second weight included in the weight matrix and a second portion of the input feature map corresponding to the second weight to the processing element.

5. The neural network operation apparatus of claim 1 , wherein the controller is configured to:
map a first weight in the weight matrix and a first portion of the input feature map corresponding to the first weight in a channel direction of the input feature map; and
map a second weight and a second portion of the input feature map corresponding to the second weight in a channel direction of an output feature map.

6. The neural network operation apparatus of claim 5 , wherein the controller is configured to alternately perform the mapping in the channel direction of the input feature map and the mapping in the channel direction of the output feature map.

7. The neural network operation apparatus of claim 1 , wherein the controller is configured to reorder the weight matrix based on whether the weight matrix has variable latency.

8. The neural network operation apparatus of claim 7 , wherein the controller is configured to:
generate a binary matrix based on whether the weight matrix has variable latency,
reorder the binary matrix based on column elements included in the binary matrix, and
reorder the weight matrix based on the reordered binary matrix.

9. The neural network operation apparatus of claim 1 , further comprising:
an adder tree configured to add an output of the processing element; and
an accumulator configured to accumulate an output of the adder tree.

10. A neural network operation method, comprising:
storing an input feature map;
mapping a portion of the input feature map and a portion of a weight matrix to a processing element performing an operation based on the input feature map and the weight matrix; and
performing the operation based on the mapped portion of the input feature map and the mapped portion of the weight matrix,
wherein the processing element is included in a processing element array, and
a size of the processing element array is determined based on a height of an output feature map, a width of the output feature map, and a size of the weight matrix.

11. The neural network operation method of claim 10 , wherein the performing comprises:
multiplexing the portion of the input feature map; and
shifting the multiplexed portion of the input feature map based on the portion of the weight matrix.

12. The neural network operation method of claim 10 , wherein the processing element array comprises two-dimensional (2D) processing element planes, and
a size of each of the processing element planes is determined based on the height of the output feature map and the width of the output feature map.

13. The neural network operation method of claim 10 , wherein the mapping comprises:
mapping a first weight included in the weight matrix and a first portion of the input feature map corresponding to the first weight to the processing element; and
mapping a second weight included in the weight matrix and a second portion of the input feature map corresponding to the second weight to the processing element.

14. The neural network operation method of claim 10 , wherein the mapping comprises:
mapping a first weight in the weight matrix and a first portion of the input feature map corresponding to the first weight in a channel direction of the input feature map; and
mapping a second weight and a second portion of the input feature map corresponding to the second weight in a channel direction of an output feature map.

15. The neural network operation method of claim 14 , wherein the mapping of comprises alternately performing the mapping in the channel direction of the input feature map and the mapping in the channel direction of the output feature map.

16. The neural network operation method of claim 10 , further comprising:
reordering the weight matrix based on whether the weight matrix has variable latency.

17. The neural network operation method of claim 16 , wherein reordering the weight matrix comprises:
generating a binary matrix based on whether the weight matrix has variable latency;
reordering the binary matrix based on column elements included in the binary matrix; and
reordering the weight matrix based on the reordered binary matrix.

18. The neural network operation method of claim 10 , further comprising:
adding an output of the processing element; and
accumulating the added output of the processing element.

Assignees

• Samsung Electronics Co Ltd
• Ulsan Nat Inst Science & Tech Unist

Inventors

• Lee Sehwan
• Lee Jongeun
• Choi Jooyeon

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F7/78

  for changing the order of data flow, e.g. matrix transposition or LIFO buffers; Overflow or underflow handling therefor · CPC title

• G06F7/762

  having at least two separately controlled rearrangement levels, e.g. multistage interconnection networks (G06F7/764 - G06F7/768 take precedence) · CPC title

• G06F17/15

  Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)} · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title