System cache optimizations for deep learning compute engines

Patent metadata

Field	Value
Publication number	US-11003592-B2
Application number	US-201715494922-A
Country	US
Kind code	B2
Filing date	Apr 24, 2017
Priority date	Apr 24, 2017
Publication date	May 11, 2021
Grant date	May 11, 2021

Abstract

In an example, an apparatus comprises a plurality of compute engines; and logic, at least partially including hardware logic, to detect a cache line conflict in a last-level cache (LLC) communicatively coupled to the plurality of compute engines; and implement context-based eviction policy to determine a cache way in the cache to evict in order to resolve the cache line conflict. Other embodiments are also disclosed and claimed.

First claim

The invention claimed is:

1. An apparatus comprising:
a plurality of compute engines communicatively coupled to a last-level cache (LLC) by an interconnect, the LLC configured with a variable cache line size, wherein the plurality of compute engines execute one or more layers of a deep learning (DL) network using the LLC; and
a controller to:
assign one or more of the plurality of compute engines as clients of the LLC;
assign a context identifier (ID) to the clients of the LLC;
receive, from the plurality of compute engines, a plurality of cache access requests, each cache access request comprising:
the context ID corresponding to the compute engine which originated the cache access request; and
metadata that indicates a total size of data to be accessed in one or more subsequent data access transactions;

detect a cache line conflict in the LLC between a first cache access request from a first compute engine and a second cache access request from a second compute engine;
responsive to detecting the cache line conflict, implement a context-based cache eviction policy that utilizes the context ID of the first compute engine and the context ID of the second compute engine to determine a cache way in the LLC to evict in order to resolve the cache line conflict, wherein the LLC is dynamically configured with partitions corresponding to each of the clients based on the one or more layers of the DL network that each of the clients execute; and
responsive to a cache miss, select a variable cache line size to allocate and fill from the LLC, wherein the variable cache line size is selected based on the total size of the data indicated in the metadata received with the access request, and based on data utilization of the LLC by the DL network.

2. The apparatus of claim 1 , wherein:
the cache eviction policy is a function of the context identifier.

3. The apparatus of claim 1 , wherein:
the LLC is reconfigured dynamically into a plurality of individually addressable caches.

4. The apparatus of claim 3 , wherein:
the LLC is reconfigured with a variable cache size.

5. An electronic device, comprising:
a general purpose graphics processor;
a plurality of compute engines communicatively coupled to a last-level cache (LLC) by an interconnect, the LLC configured with a variable cache line size, wherein the plurality of compute engines execute one or more layers of a deep learning (DL) network using the LLC; and
a controller communicably coupled to the general purpose graphics processor and the plurality of compute engines, the controller to:
assign one or more of the plurality of compute engines as clients of the LLC;
assign a context identifier (ID) to the clients of the LLC;
receive, from the plurality of compute engines, a plurality of cache access requests, each cache access request comprising:
the context ID corresponding to the compute engine which originated the cache access request; and
metadata that indicates a total size of data to be accessed in one or more subsequent data access transactions;

detect a cache line conflict in the LLC between a first cache access request from a first compute engine and a second cache access request from a second compute engine;
responsive to detecting the cache line conflict, implement a context-based cache eviction policy that utilizes the context ID of the first compute engine and the context ID of the second compute engine to determine a cache way in the LLC to evict in order to resolve the cache line conflict, wherein the LLC is dynamically configured with partitions corresponding to each of the clients based on the one or more layers of the DL network that each of the clients execute; and
responsive to a cache miss, select a variable cache line size to allocate and fill from the LLC, wherein the variable cache line size is selected based on the total size of the data indicated in the metadata received with the access request, and based on data utilization of the LLC by the DL network.

6. The electronic device of claim 5 , wherein:
the cache eviction policy is a function of the context identifier.

7. The electronic device of claim 5 wherein:
the LLC is reconfigured dynamically into a plurality of individually addressable caches.

8. The electronic device of claim 7 , wherein:
the LLC is reconfigured with a variable cache size.

9. One or more non-transitory computer-readable medium comprising one or more instructions that when executed on at least one processor configure the at least one processor to perform one or more operations to:
assign one or more of a plurality of compute engines communicatively coupled to a last-level cache (LLC) by an interconnect as clients of the LLC, the LLC configured with a variable cache line size, wherein the plurality of compute engines execute one or more layers of a deep learning (DL) network using the LLC;
assign a context identifier (ID) to the clients of the LLC;
receive, from the plurality of compute engines, a plurality of cache access requests, each cache access request comprising:
the context ID corresponding to the compute engine which originated the cache access request; and
metadata that indicates a total size of data to be accessed in one or more subsequent data access transactions;

detect a cache line conflict in the LLC between a first cache access request from a first compute engine and a second cache access request from a second compute engine;
responsive to detecting the cache line conflict, implement a context-based cache eviction policy that utilizes the context ID of the first compute engine and the context ID of the second compute engine to determine a cache way in the LLC to evict in order to resolve the cache line conflict, wherein the LLC is dynamically configured with partitions corresponding to each of the clients based on the one or more layers of the DL network that each of the clients execute; and
responsive to a cache miss, select a variable cache line size to allocate and fill from the LLC, wherein the variable cache line size is selected based on the total size of the data indicated in the metadata received with the access request, and based on data utilization of the LLC by the DL network.

10. The one or more non-transitory computer-readable medium of claim 9 , wherein:
the cache eviction policy is a function of the context identifier.

11. The one or more non-transitory computer-readable medium of claim 9 , wherein:
the LLC is reconfigured dynamically into a plurality of individually addressable caches.

12. The one or more non-transitory computer-readable medium of claim 11 , wherein:
the LLC is reconfigured with a variable cache size.

13. A method comprising:
assigning one or more of a plurality of compute engines communicatively coupled to a last-level cache (LLC) by an interconnect as clients of the LLC, the LLC configured with a variable cache line size, wherein the plurality of compute engines execute one or more layers of a deep learning (DL) network using the LLC;
assigning a context identifier (ID) to the clients of the LLC;
receiving, from the plurality of compute engines, a plurality of cache access requests, each cache access request comprising:
the context ID corresponding to the compute engine which originated the cache access request; and
metadata that indicates a total size of data to be accessed in one or more subsequent data access transactions;

detecting a cache line conflict in the LLC between a first cache access request from a first compute engine and a second cache access request from a second compute engine;
responsive to detecting the cache line conflict, implementing

Assignees

• Intel Corp

Inventors

• Zmora Neta
• Ben-Avi Eran

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title