Mapping convolution to a channel convolution engine

Patent metadata

Field	Value
Publication number	US-11537865-B2
Application number	US-202016793961-A
Country	US
Kind code	B2
Filing date	Feb 18, 2020
Priority date	Feb 18, 2020
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

A processor system comprises a first and second group of registers and a hardware channel convolution processor unit. The first group of registers is configured to store data elements of channels of a portion of a convolution data matrix. Each register stores at least one data element from each channel. The second group of registers is configured to store data elements of convolution weight matrices including a separate convolution weight matrix for each channel. Each register stores at least one data element from each convolution weight matrix. The hardware channel convolution processor unit is configured to multiply each data element in the first group of registers with a corresponding data element in the second group of registers and sum together the multiplication results for each specific channel to determine corresponding channel convolution result data elements in a corresponding channel convolution result matrix.

First claim

What is claimed is:

1. A processor system, comprising:
a first group of registers configured to store data elements of a plurality of channels of a portion of a convolution data matrix, wherein each register of the first group of registers stores at least one data element from each of the plurality of channels;
a second group of registers configured to store data elements of a plurality of convolution weight matrices including a separate convolution weight matrix for each of the plurality of channels, wherein each register of the second group of registers stores at least one data element from each of the plurality of convolution weight matrices; and
a hardware channel convolution processor unit configured to:
for each data element in the first group of registers, multiply the data element in the first group of registers with a corresponding data element in the second group of registers to determine a corresponding multiplication result in multiplication results; and
for each specific channel of the plurality of channels, sum together ones of the multiplication results corresponding to the specific channel to determine at least one corresponding channel convolution result data element in a corresponding channel convolution result matrix;

wherein the first group of registers is configured to retain data elements of the portion of the convolution data matrix across the plurality of channels from a first convolution processing iteration that overlap with data elements of a different portion of the convolution data matrix across the plurality of channels for a second convolution processing iteration.

2. The system of claim 1 , wherein a total count of the stored data elements of the first group of registers is the same as a total count of the stored data elements of the second group of registers.

3. The system of claim 1 , wherein the hardware channel convolution processor unit comprises a plurality of calculation units and each calculation unit of the plurality of calculation units is configured to receive a plurality of data elements of the first group of registers corresponding to a same channel of the convolution data matrix and a plurality of corresponding data elements of the second group of registers corresponding to the separate convolution weight matrix for the same channel of the convolution data matrix.

4. The system of claim 3 , wherein each calculation unit of the plurality of calculation units includes a different vector multiply unit and a different vector adder unit.

5. The system of claim 4 , wherein each of the different vector adder units includes a different adder tree.

6. The system of claim 1 , wherein the convolution data matrix is a three-dimensional machine learning data matrix.

7. The system of claim 1 , further comprising a data input unit configured to: process the data elements stored in the first group of registers by channel into a plurality of data input vectors, wherein each of the plurality of data input vectors includes data elements corresponding to a two-dimensional sub-matrix of the convolution data matrix.

8. The system of claim 1 , further comprising a weight input unit configured to: process the data elements stored in the second group of registers into a plurality of weight input vectors, wherein each of the plurality of weight input vectors includes data elements corresponding to one of the plurality of convolution weight matrices.

9. The system of claim 1 , wherein each of the plurality of convolution weight matrices is a 3×3, 5×5, 7×7, 9×9, or 11×11 matrix.

10. The system of claim 1 , wherein the data elements stored in the first group of registers are 4-bit, 8-bit, 2-byte, or 4-byte elements.

11. The system of claim 1 , wherein a total count of the stored data elements of each of the first group of registers is a multiple of a cache line size.

12. A method, comprising:
receiving a convolution operation instruction specifying a convolution data matrix and a set of convolution weight matrices;
assigning a different portion of the convolution data matrix to each of a plurality of processing elements;
transmitting a plurality of data elements corresponding to the different assigned portion of the convolution data matrix to each of the plurality of processing elements;
broadcasting to each of the plurality of processing elements assigned a same channel of the convolution data matrix a same subset of the set of convolution weight matrices;
receiving from the plurality of processing elements channel convolution result data elements of a channel convolution result matrix determined using hardware channel convolution processor units of the plurality of processing elements; and
storing the channel convolution result matrix to a memory location;
wherein at least one of the plurality of processing elements stores in a first group of registers data elements of a plurality of channels of a portion of the convolution data matrix, wherein each register of the first group of registers stores at least one data element from each of the plurality of channels, and the first group of registers is configured to retain data elements of the portion of the convolution data matrix across the plurality of channels from a first convolution processing iteration that overlap with data elements of a different portion of the convolution data matrix across the plurality of channels for a second convolution processing iteration.

13. The method of claim 12 , wherein the convolution data matrix and the channel convolution result matrix are stored using a channel-first layout format.

14. The method of claim 12 , wherein the convolution data matrix is a three-dimensional machine learning data matrix and each of the set of convolution weight matrices is a two-dimensional matrix.

15. The method of claim 12 , wherein the at least one of the plurality of processing elements:
stores in a second group of registers data elements of a subset of the set of convolution weight matrices including a separate convolution weight matrix for each of the plurality of channels, wherein each register of the second group of registers stores at least one data element from each of the subset of the set of convolution weight matrices.

16. The method of claim 15 , wherein the at least one of the hardware channel convolution processor units of the plurality of processing elements:
for each data element in the first group of registers, multiplies the data element in the first group of registers with a corresponding data element in the second group of registers to determine a corresponding multiplication result in multiplication results; and
for each specific channel of the plurality of channels, sums together ones of the multiplication results corresponding to the specific channel to determine one corresponding channel convolution result data element in the corresponding channel convolution result matrix.

17. The method of claim 16 , wherein the at least one of the hardware channel convolution processor units of the plurality of processing elements comprises a plurality of calculation units and each calculation unit receives a plurality of data elements of the first group of registers corresponding to a same channel of the convolution data matrix and a plurality of corresponding data elements of the second group of registers corresponding to the separate convolution weight matrix for the same channel of the convolution data matrix.

18. A method, comprising:
storing at a hardware processing element in a first group of registers data elements of a plurality of channels of a f

Assignees

• Meta Platforms Inc

Inventors

• Nair Krishnakumar Narayanan
• Komuravelli Rakesh
• Diril Abdulkadir Utku
• Zadeh Ehsan Khish Ardestani
• Hao Yuchen
• Schatz Martin
• Ulrich Thomas Mark
• Wu Olivia
• Kadkol Anup Ramesh
• Firoozshahian Amin

Classifications

• G06F9/3013 Primary

  according to data content, e.g. floating-point registers, address registers · CPC title

• G06F17/153

  Multidimensional correlation or convolution · CPC title

• G06N3/063

  using electronic means · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• G06F9/3001

  Arithmetic instructions · CPC title