Systolic array having support for output sparsity

Patent metadata

Field	Value
Publication number	US-12399685-B2
Application number	US-202117304803-A
Country	US
Kind code	B2
Filing date	Jun 25, 2021
Priority date	Jun 25, 2021
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

A processing apparatus is described herein that includes a general-purpose parallel processing engine comprising a matrix accelerator including one or more systolic arrays, at least one of the one or more systolic arrays comprising multiple pipeline stages, each pipeline stage of the multiple pipeline stages including multiple processing elements, the multiple processing elements configured to perform processing operations on input matrix elements based on output sparsity metadata. The output sparsity metadata indicates to the multiple processing elements to bypass multiplication for a first row of elements of a second matrix and multiply a second row of elements of the second matrix with a column of matrix elements of a first matrix.

First claim

What is claimed is:

1. A processing apparatus including:
a general-purpose parallel processing engine comprising a matrix accelerator including one or more systolic arrays, at least one of the one or more systolic arrays comprising multiple pipeline stages, each pipeline stage of the multiple pipeline stages including multiple processing elements, the multiple processing elements configured to:
perform processing operations on input matrix elements based on output sparsity metadata, the output sparsity metadata to indicate to the multiple processing elements to bypass multiplication for a first row of elements of a second matrix and multiply a second row of elements of the second matrix with a column of matrix elements of a first matrix,
wherein the output sparsity metadata is independent of input sparsity of the input matrix elements, the multiple processing elements are to generate a sparse output matrix in a compact format, and the output sparsity metadata is to enable de-compaction of the sparse output matrix via insertion of a zero value for a bypassed row of elements.

2. The processing apparatus as in claim 1 , comprising circuitry to decode an instruction into a decoded instruction, the decoded instruction including multiple sub-instructions.

3. The processing apparatus as in claim 1 , wherein the multiple processing elements include a processing element having a first source input associated with an accumulator value, a second source input associated with the first matrix, and multiple third source inputs associated with the second matrix.

4. The processing apparatus as in claim 3 , wherein the output sparsity metadata is to indicate which of the multiple third source inputs to multiply with the second source input.

5. The processing apparatus as in claim 4 , wherein the output sparsity metadata is to indicate to multiply at least one of the multiple third source inputs.

6. The processing apparatus as in claim 5 , wherein the multiple third source inputs include three third source inputs and the output sparsity metadata includes a bit associated with each of the three third source inputs.

7. The processing apparatus as in claim 5 , wherein the processing element is associated with a first processing channel and each pipeline stage of the multiple pipeline stages includes multiple processing channels.

8. The processing apparatus as in claim 1 , wherein, in a first processing cycle, the output sparsity metadata is to indicate to a first processing element to multiply the second row of elements of the second matrix and bypass multiplication of the first row of elements of the second matrix and a third row of elements of the second matrix and, based on the output sparsity metadata, the first processing element is to multiply the second row of elements of the second matrix with the column of matrix elements of the first matrix.

9. The processing apparatus as in claim 8 , wherein, in a second processing cycle, the output sparsity metadata is to indicate to the first processing element to bypass multiplication of the third row of elements of the second matrix and to multiply each of the second row of elements of the second matrix and a fourth row of elements of the second matrix with the column of matrix elements of the first matrix, and, based on the output sparsity metadata and in response to a determination that the second row of elements of the second matrix was multiplied in the first processing cycle, the first processing element is to multiply the fourth row of elements of the second matrix with the column of matrix elements of the first matrix.

10. A method comprising:
fetching an instruction at a processing resource of a graphics processor to perform operations associated with a matrix instruction that specifies metadata for output sparsity;
decoding the instruction into a decoded instruction;
reading operand data for the decoded instruction from a register file of the processing resource, the operand data including matrix elements and the metadata for output sparsity;
executing the decoded instruction via a matrix accelerator by performing multiply-accumulate operations on matrix elements selected based on the metadata for output sparsity; and
writing output of the multiply-accumulate operations to the register file, wherein the metadata for output sparsity is independent of input sparsity of the matrix elements, output of the multiply-accumulate operations includes a sparse output matrix in a compact format, and the metadata for output sparsity enables de-compaction of the sparse output matrix via insertion of a zero value for a bypassed row of elements.

11. The method as in claim 10 , comprising, in a first processing cycle based on the metadata for output sparsity:
multiplying via first processing element, a second row of elements of a second matrix and bypassing multiplication of a first row of elements of the second matrix and a third row of elements of the second matrix; and
multiplying the second row of elements of the second matrix with a column of matrix elements of a first matrix.

12. The method as in claim 10 , wherein decoding the instruction into the decoded instruction includes decoding the instruction into multiple sub-instructions and executing the decoded instruction includes executing the multiple sub-instructions.

13. The method as in claim 10 , wherein executing the decoded instruction via the matrix accelerator includes:
receiving a first set of matrix elements at a first processing element of the matrix accelerator;
receiving multiple second sets of matrix elements at the first processing element of the matrix accelerator;
selecting a second set of matrix elements from the multiple second sets of matrix elements using the metadata for output sparsity; and
performing a multiply-accumulate operation at the first processing element on the first set of matrix elements and the second set of matrix elements selected using the metadata for output sparsity.

14. The method as in claim 13 , further comprising outputting a result of the multiply-accumulate operation and the metadata for output sparsity to a second processing element of the matrix accelerator.

15. The method as in claim 14 , wherein the matrix accelerator includes a systolic array including multiple pipeline stages, the first processing element is associated with a first pipeline stage, and the second processing element is associated with a second pipeline stage.

16. A system comprising:
a memory device; and
a graphics processor coupled to the memory device, the graphics processor comprising a general-purpose parallel processing engine including:
a matrix accelerator including one or more systolic arrays, at least one of the one or more systolic arrays comprising multiple pipeline stages, each pipeline stage of the multiple pipeline stages including multiple processing elements, the multiple processing elements configured to:
perform processing operations on input matrix elements based on output sparsity metadata, the output sparsity metadata to indicate to the multiple processing elements to bypass multiplication for a first row of elements of a second matrix and multiply a second row of elements of the second matrix with a column of matrix elements of a first matrix,
wherein the output sparsity metadata is independent of input sparsity of the input matrix elements, the multiple processing elements are to generate a sparse output matrix in a compact format, and the output sparsity metadata is to enable de-compaction of the sparse output matrix via insertion of a zero value for a bypassed row of elem

Assignees

• Intel Corp

Inventors

• Parra Jorge
• Fu Fangwen
• Maiyuran Subramaniam
• George Varghese
• Macpherson Mike
• Pal Supratim
• Gurram Chandra
• Ganapathy Sabareesh
• Avancha Sasikanth
• Vooturi Dharma Teja
• Mellempudi Naveen
• Das Dipankar

Classifications

• G06F9/30036

  Instructions to perform operations on packed data, e.g. vector, tile or matrix operations · CPC title

• G06F9/30038

  using a mask · CPC title

• G06F17/16 Primary

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title

• G06F15/8046

  Systolic arrays · CPC title

• G06F7/523

  Multiplying only · CPC title