Mapping convolution to a partition channel convolution engine

Patent metadata

Field	Value
Publication number	US-11520853-B2
Application number	US-202016805339-A
Country	US
Kind code	B2
Filing date	Feb 28, 2020
Priority date	Feb 28, 2020
Publication date	Dec 6, 2022
Grant date	Dec 6, 2022

Abstract

A processor system comprises two groups of registers and a hardware channel convolution processor unit. The first group of registers is configured to store data elements of channels of a portion of a convolution data matrix. Each register stores at least one data element from each channel. The second group of registers is configured to store data elements of convolution weight matrices including a separate matrix for each channel. Each register stores at least one data element from each matrix. The hardware channel convolution processor unit is configured to multiply each data element in a first and second portion of the first group of registers with a corresponding data element in the second group of registers to determine corresponding multiplication results and sum together the multiplication results for each specific channel to determine two corresponding channel convolution result data elements in a corresponding channel convolution result matrix.

First claim

What is claimed is:

1. A processor system, comprising:
a first group of registers configured to store data elements of a plurality of channels of a portion of a convolution data matrix, wherein each register of the first group of registers stores at least one data element from each of the plurality of channels;
a second group of registers configured to store data elements of a plurality of convolution weight matrices including a separate convolution weight matrix for each of the plurality of channels, wherein each register of the second group of registers stores at least one data element from each of the plurality of convolution weight matrices; and
a hardware processor unit configured to:
for each data element in a first portion of the first group of registers, multiply the data element in the first portion with a corresponding data element in the second group of registers to determine a corresponding multiplication result in first multiplication results, wherein the first portion corresponds to a first sub matrix of the convolution data matrix;
for each data element in a second portion of the first group of registers, multiply the data element in the second portion with a corresponding data element in the second group of registers to determine a corresponding multiplication result in second multiplication results, wherein the second portion corresponds to a second sub matrix of the convolution data matrix that is different from the first sub matrix but the second sub matrix at least in part overlaps with the first sub matrix of the convolution data matrix; and
for each specific channel of the plurality of channels, sum together ones of the first multiplication results corresponding to the specific channel to determine one corresponding channel convolution result data element in a corresponding channel convolution result matrix and sum together ones of the second multiplication results corresponding to the specific channel to determine another one corresponding channel convolution result data element in the corresponding channel convolution result matrix.

2. The system of claim 1 , wherein a total count of the stored data elements of the first group of registers is greater than a total count of the stored data elements of the second group of registers.

3. The system of claim 1 , wherein the hardware processor unit is configured to determine the first multiplication results and the second multiplication results at least in part concurrently.

4. The system of claim 1 , wherein the hardware processor unit is configured to determine channel convolution result data elements associated with the first portion of the first group of registers and the second portion of the first group of registers at least in part concurrently.

5. The system of claim 1 , wherein the data elements in the first portion of the first group of registers match at least two-thirds of the data elements in the second portion of the first group of registers.

6. The system of claim 1 , wherein the first portion of the first group of registers overlaps with the second portion of the first group of registers and the second portion of the first group of registers includes a group of data elements that are different from the data elements in the first portion of the first group of registers.

7. The system of claim 1 , wherein the hardware processor unit is configured to receive a plurality of data elements of the first group of registers corresponding to a same channel of the convolution data matrix and a plurality of corresponding data elements of the second group of registers corresponding to the separate convolution weight matrix for the same channel of the convolution data matrix.

8. The system of claim 7 , wherein the hardware processor unit includes a plurality of vector units, each vector unit of the plurality of vector units includes a different vector multiply unit and a different vector adder unit.

9. The system of claim 8 , wherein each of the different vector adder units includes a different adder tree.

10. The system of claim 1 , wherein the convolution data matrix is a three-dimensional machine learning data matrix.

11. The system of claim 1 , wherein the hardware processor unit is further configured to: process the data elements stored in the first group of registers by channel into a plurality of data input vectors, wherein each of the plurality of data input vectors includes data elements corresponding to a two-dimensional sub-matrix of the convolution data matrix.

12. The system of claim 1 , wherein the hardware processor unit is further configured to: process the data elements stored in the second group of registers into a plurality of weight input vectors, wherein each of the plurality of weight input vectors includes data elements corresponding to one of the plurality of convolution weight matrices.

13. The system of claim 1 , wherein each of the plurality of convolution weight matrices is a 3×3, 5×5, 7×7, 9×9, or 11×11 matrix.

14. The system of claim 1 , wherein the data elements stored in the first group of registers are 4-bit, 8-bit, 2-byte, or 4-byte elements.

15. The system of claim 1 , wherein a total count of the stored data elements of each of the first group of registers is a multiple of a cache line size.

16. A method, comprising:
storing at a hardware processing element in a first group of registers data elements of a plurality of channels of a first portion of a convolution data matrix, wherein each register of the first group of registers stores at least one data element from each of the plurality of channels;
storing at the hardware processing element in a second group of registers data elements of a subset of a set of convolution weight matrices including a separate convolution weight matrix for each of the plurality of channels, wherein each register of the second group of registers stores at least one data element from each of the subset of the set of convolution weight matrices;
for each data element in a first portion of the first group of registers, multiplying the data element in the first portion with a corresponding data element in the second group of registers to determine a corresponding multiplication result in first multiplication results, wherein the first portion corresponds to a first sub matrix of the convolution data matrix;
for each data element in a second portion of the first group of registers, multiplying the data element in the second portion with a corresponding data element in the second group of registers to determine a corresponding multiplication result in second multiplication results, wherein the second portion corresponds to a second sub matrix of the convolution data matrix that is different from the first sub matrix but the second sub matrix at least in part overlaps with the first sub matrix of the convolution data matrix;
for each specific channel of the plurality of channels, summing together ones of the first multiplication results corresponding to the specific channel to determine one corresponding channel convolution result data element in a corresponding channel convolution result matrix; and
for each specific channel of the plurality of channels, summing together ones of the second multiplication results corresponding to the specific channel to determine another one corresponding channel convolution result data element in the corresponding channel convolution result matrix.

17. The method of claim 16 , further comprising:
saving in the first group of registers data elements that overlap between the first portion of the convolution data matrix

Assignees

• Meta Platforms Inc

Inventors

• Nair Krishnakumar Narayanan
• Komuravelli Rakesh
• Diril Abdulkadir Utku
• Zadeh Ehsan Khish Ardestani
• Hao Yuchen
• Schatz Martin
• Ulrich Thomas Mark
• Wu Olivia
• Kadkol Anup Ramesh
• Firoozshahian Amin

Classifications

• G06F17/16 Primary

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• G06F17/153 Primary

  Multidimensional correlation or convolution · CPC title

• G06F18/2414

  Smoothing the distance, e.g. radial basis function networks [RBFN] · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06F9/30036

  Instructions to perform operations on packed data, e.g. vector, tile or matrix operations · CPC title