Utilizing structured sparsity in systolic arrays

Patent metadata

Field	Value
Publication number	US-12405787-B2
Application number	US-202418621539-A
Country	US
Kind code	B2
Filing date	Mar 29, 2024
Priority date	Nov 30, 2020
Publication date	Sep 2, 2025
Grant date	Sep 2, 2025

Abstract

An apparatus to facilitate utilizing structured sparsity in systolic arrays is disclosed. The apparatus includes a processor comprising a systolic array to receive data from a plurality of source registers, the data comprising unpacked source data, structured source data that is packed based on sparsity, and metadata corresponding to the structured source data; identify portions of the unpacked source data to multiply with the structured source data, the portions of the unpacked source data identified based on the metadata; and output, to a destination register, a result of multiplication of the portions of the unpacked source data and the structured source data.

First claim

What is claimed is:

1. An apparatus comprising:
a processor comprising a systolic array to:
execute an instruction for sparse systolic dot product accumulate;
read at least portions of elements of a plurality of source registers referenced by the instruction, wherein the plurality of source registers comprise a first source register having metadata corresponding to structured source data, a second source register having unpacked source data, and a third source register having the structured source data packed based on sparsity as packed source data;
provide a first subset of elements of the packed source data to at least one stage of the systolic array, the at least one stage comprising dot product circuitry;
select, using the metadata, a second subset of elements of the unpacked source data to utilize the at least one stage of the systolic array, the second subset of elements corresponding to the first subset of elements; and
perform, at the at least one stage of the systolic array, dot product accumulate operations.

2. The apparatus of claim 1 , wherein the systolic array to perform the dot product accumulate operations using the first subset of elements and the second subset of elements.

3. The apparatus of claim 1 , wherein the systolic array further comprises a plurality of multiplexor circuits to select the portions of the unpacked source data to multiply with the structured source data based on the metadata.

4. The apparatus of claim 1 , wherein the structured source data that is packed based on the sparsity comprises elements of a broadcast register of the systolic array.

5. The apparatus of claim 1 , wherein the structured source data that is packed based on the sparsity comprises elements of an index register of the systolic array.

6. The apparatus of claim 1 , wherein the systolic array to execute the instruction for sparse systolic dot product accumulate in order to identify the portions of the unpacked source data to multiply with the structured source data using the metadata and to perform a dot product multiplication of the portions with a result of multiplication of the structured source data, and wherein the metadata is provided in a source register of the plurality of source registers called by the instruction.

7. The apparatus of claim 1 , wherein the packed source data comprises at least one of a half-float datatype that packs two 16-bit elements into a channel, a bfloat datatype that packs two 16-bit elements into a channel, an int8 datatype that packs four 8-bit elements into a channel, an int4 datatype that packs eight 4-bit elements into a channel, or an int2 datatype that packs sixteen 2-bit elements into a channel.

8. The apparatus of claim 1 , wherein the metadata indicates a position of non-zero elements in an original form of the structured source data prior to packing into the structured source data.

9. The apparatus of claim 8 , wherein the original form of the structured source data is pre-processed by an external agent to pack into the structured source data by removing sparse elements from the original form, and wherein the external agent generates the metadata.

10. The apparatus of claim 9 , wherein the external agent comprises at least one of a central processing unit (CPU) or an intelligent sensor.

11. The apparatus of claim 1 , wherein the processor comprises a general-purpose graphics processing unit (GPGPU).

12. At least one non-transitory machine readable storage medium comprising instructions that, when executed, cause at least one processor to perform operations comprising:
executing an instruction for sparse systolic dot product accumulate;
reading at least portions of elements of a plurality of source registers referenced by the instruction, wherein the plurality of source registers comprise a first source register having metadata corresponding to structured source data, a second source register having unpacked source data, and a third source register having the structured source data packed based on sparsity as packed source data;
providing a first subset of elements of the packed source data to at least one stage of a systolic array, the at least one stage comprising dot product circuitry;
selecting, using the metadata, a second subset of elements of the unpacked source data to utilize the at least one stage of the systolic array, the second subset of elements corresponding to the first subset of elements; and
performing, at the at least one stage of the systolic array, dot product accumulate operations.

13. The at least one non-transitory machine readable storage medium of claim 12 , wherein the systolic array to perform the dot product accumulate operations using the first subset of elements and the second subset of elements.

14. The at least one non-transitory machine readable storage medium of claim 12 , wherein the systolic array further comprises a plurality of multiplexor circuits to select the portions of the unpacked source data to multiply with the structured source data based on the metadata.

15. The at least one non-transitory machine readable storage medium of claim 12 , wherein the systolic array to execute the instruction for sparse systolic dot product accumulate in order to identify the portions of the unpacked source data to multiply with the structured source data using the metadata and to perform a dot product multiplication of the portions with a result of multiplication of the structured source data, and wherein the metadata is provided in a source register of the plurality of source registers called by the instruction.

16. The at least one non-transitory machine readable storage medium of claim 12 ,
wherein the metadata indicates a position of non-zero elements in an original form of the structured source data prior to packing into the structured source data.

17. A method comprising:
executing by a systolic array of a processing device, an instruction for sparse systolic dot product accumulate;
reading at least portions of elements of a plurality of source registers referenced by the instruction, wherein the plurality of source registers comprise a first source register having metadata corresponding to structured source data, a second source register having unpacked source data, and a third source register having the structured source data packed based on sparsity as packed source data;
providing a first subset of elements of the packed source data to at least one stage of the systolic array, the at least one stage comprising dot product circuitry;
selecting, using the metadata, a second subset of elements of the unpacked source data to utilize the at least one stage of the systolic array, the second subset of elements corresponding to the first subset of elements; and
performing, at the at least one stage of the systolic array, dot product accumulate operations.

18. The method of claim 17 , wherein the systolic array to perform the dot product accumulate operations using the first subset of elements and the second subset of elements.

19. The method of claim 17 , wherein the systolic array further comprises a plurality of multiplexor circuits to select the portions of the unpacked source data to multiply with the structured source data based on the metadata.

20. The method of claim 17 , wherein the systolic array to execute the instruction for sparse systolic dot product accumulate in order to identify the portions of the unpacked source data to multiply with the structured source data using the metadata and to perform a dot product multi

Assignees

• Intel Corp

Inventors

• Maiyuran Subramaniam
• Parra Jorge
• Garg Ashutosh
• Gurram Chandra
• Mei Chunhui
• Borkar Durgesh
• Marwaha Shubra
• Pal Supratim
• George Varghese
• Xiong Wei
• Li Yan
• Liu Yongsheng
• Das Dipankar
• Avancha Sasikanth
• Vooturi Dharma Teja
• Mellempudi Naveen K

Classifications

• G06F9/3001 Primary

  Arithmetic instructions · CPC title

• G06F9/3893

  controlled in tandem, e.g. multiplier-accumulator · CPC title

• G06F15/8046

  Systolic arrays · CPC title

• G06F9/30101

  Special purpose registers · CPC title

• G06F9/30036 Primary

  Instructions to perform operations on packed data, e.g. vector, tile or matrix operations · CPC title