Method and apparatus for performing atomic operations on local cache slots of a shared global memory

Patent metadata

Field	Value
Publication number	US-11074113-B1
Application number	US-202016879831-A
Country	US
Kind code	B1
Filing date	May 21, 2020
Priority date	May 21, 2020
Publication date	Jul 27, 2021
Grant date	Jul 27, 2021

Abstract

A storage system includes at least two independent storage engines interconnected by a fabric, each storage engine having two compute nodes. A shared global memory is implemented using cache slots of each of the compute nodes. Memory access operations to the slots of shared global memory are managed by a fabric adapter to guarantee that the operations are atomic. To enable local cache operations to be managed independent of the fabric adapter, a cache metadata data structure includes a global flag bit for each cache slot, that is used to designate the cache slot as globally available or temporarily reserved for local IO processing. The cache metadata data structure also includes a mutex (Peterson lock) for each cache slot to enforce a mutual exclusion concurrency control policy on the cache slot between the two compute nodes of the storage engine when the cache slot is used for local IO processing.

First claim

What is claimed is:

1. A storage system, comprising:
a first storage engine having a first compute node, a second compute node, a first fabric adapter, and a second fabric adapter, the first compute node having a first cache and the second compute node having a second cache;
a second storage engine having a third compute node, a fourth compute node, a third fabric adapter, and a fourth fabric adapter, the third compute node having a third cache and the fourth compute node having a fourth cache;
the first cache, second cache, third cache, and fourth cache each containing respective cache slots, and at least a subset of the cache slots forming a shared global memory for use by the first, second, third and fourth compute nodes; and
a cache metadata data structure having an entry for each of the cache slots forming the shared global memory, each entry including a global flag bit and a mutex, the global flag bit identifying a respective cache slot as either being managed by one of the fabric adapters or not managed by the one of the fabric adapters, and the mutex being configured to enforce a mutual exclusion concurrency control policy on the cache slot when the cache slot is not managed by the one of the fabric adapters.

2. The storage system of claim 1 , wherein the mutex is a Peterson lock.

3. The storage system of claim 1 , further comprising control logic configured to:
receive an IO operation by the storage system;
provide the IO operation to the first compute node of the first storage engine;
select, by the first compute node, a cache slot from the shared global memory to be used to process the IO operation;
if the selected cache slot is not located within the first storage engine, manage the IO operation by the first fabric adapter of the first compute node; and
if the selected cache slot is located within the first storage engine:
set the global flag bit for the selected cache slot in the cache metadata data structure;
obtain control of the mutex for the selected cache slot in the cache metadata data structure; and
process the IO operation against the selected cache slot without use of the first fabric adapter of the first compute node.

4. The storage system of claim 3 , wherein the IO operation is a write operation.

5. The storage system of claim 4 , wherein the control logic configured to process the IO operation comprises control logic configured to write data to the selected cache slot and destage the data from the selected cache slot to a location in permanent storage.

6. The storage system of claim 3 , wherein the IO operation is a read operation.

7. The storage system of claim 3 , further comprising control logic configured to release control of the mutex for the selected cache slot after processing the IO operation against the selected cache slot.

8. The storage system of claim 3 , further comprising control logic configured to clear the global flag bit for the selected cache slot after releasing control of the mutex for the selected cache slot.

9. A non-transitory tangible computer readable storage medium having stored thereon a computer program for performing atomic operations on cache slots of a shared global memory, the computer program including a set of instructions which, when executed by a computer, cause the computer to perform a method comprising the steps of:
receiving an IO operation by a storage system, the storage system having at least two independent storage engines interconnected by a fabric, each storage engine having two compute nodes, each compute node having a cache with a plurality of cache slots and a fabric access module controlling access to the cache slots, wherein the fabric access modules are configured to enable at least a subset of the cache slots of each compute node to be accessible by each of the other compute nodes to form a shared global memory;
providing the IO operation to a first of the compute nodes of a first of the storage engines;
selecting, by the first of the compute nodes, a cache slot from the shared global memory to be used to process the IO operation;
if the selected cache slot is not located within the first of the storage engines, managing the IO operation by the fabric access module of the first of the compute nodes; and
if the selected cache slot is located within the first of the storage engines:
setting a global flag bit for the selected cache slot in a cache metadata data structure;
obtaining control of a mutex for the selected cache slot in the cache metadata data structure, the mutex being configured to enforce a mutual exclusion concurrency control policy on the cache slot between the two compute nodes of the first of the storage engines; and
processing the IO operation against the selected cache slot without the fabric access module of the first of the compute nodes.

10. The non-transitory tangible computer readable storage medium of claim 9 , wherein the mutex is a Peterson lock.

11. The non-transitory tangible computer readable storage medium of claim 9 , wherein the IO operation is a write operation.

12. The non-transitory tangible computer readable storage medium of claim 11 , wherein the step of processing the IO operation comprises writing data to the selected cache slot and destaging the data from the selected cache slot to a location in permanent storage.

13. The non-transitory tangible computer readable storage medium of claim 9 , wherein the IO operation is a read operation.

14. The non-transitory tangible computer readable storage medium of claim 9 , further comprising releasing control of the mutex for the selected cache slot after the step of processing the IO operation against the selected cache slot.

15. The non-transitory tangible computer readable storage medium of claim 9 , further comprising clearing the global flag bit for the selected cache slot after the step of releasing control of the mutex for the selected cache slot.

16. A method of performing atomic operations on cache slots of a shared global memory, comprising:
receiving an IO operation by a storage system, the storage system having at least two independent storage engines interconnected by a fabric, each storage engine having two compute nodes, each compute node having a cache with a plurality of cache slots and a fabric access module controlling access to the cache slots, wherein the fabric access modules are configured to enable at least a subset of the cache slots of each compute node to be accessible by each of the other compute nodes to form a shared global memory;
providing the IO operation to a first of the compute nodes of a first of the storage engines;
selecting, by the first of the compute nodes, a cache slot from the shared global memory to be used to process the IO operation;
if the selected cache slot is not located within the first of the storage engines, managing the IO operation by the fabric access module of the first of the compute nodes; and
if the selected cache slot is located within the first of the storage engines:
setting a global flag bit for the selected cache slot in a cache metadata data structure;
obtaining control of a mutex for the selected cache slot in the cache metadata data structure, the mutex being configured to enforce a mutual exclusion concurrency control policy on the cache slot between the two compute nodes of the first of the storage engines; and
processing the IO operation against the selected cache slot without the fabric access module of the first of the compute nodes.

17. The method of claim 16 , wherein the mutex is a Peterson lock.

Assignees

• Emc Ip Holding Co Llc

Inventors

• Ivester Steven
• Sahasrabudhe Kaustubh

Classifications

• G06F2212/312

  In storage controller · CPC title

• G06F2212/284

  being distributed · CPC title

• G06F2212/261

  Storage comprising a plurality of storage devices · CPC title

• G06F12/0868

  Data transfer between cache memory and other subsystems, e.g. storage devices or host systems · CPC title

• G06F12/0857 Primary

  by multiple requestors · CPC title