Use of cluster-level redundancy within a cluster of a distributed storage management system to address node-level errors

Patent metadata

Field	Value
Publication number	US-11983080-B2
Application number	US-202217680621-A
Country	US
Kind code	B2
Filing date	Feb 25, 2022
Priority date	Nov 16, 2021
Publication date	May 14, 2024
Grant date	May 14, 2024

Abstract

Systems and methods that make use of cluster-level redundancy within a distributed storage management system to address various node-level error scenarios are provided. Rather than making use of a generalized one-size-fits-all approach in an effort to reduce complexity, an approach tailored to the node-level error scenario at issue may be performed to avoid doing more than necessary. According to one embodiment, responsive to identification of a failed RAID stripe by a node of a cluster of a distributed storage management system, for each block ID of multiple block IDs associated with the failed RAID stripe, a data block is restored corresponding to the block ID by reading the data block from another node of the cluster having a redundant copy of the data block; and writing the redundant copy of the data block to a storage area of the node that is unaffected by the failed RAID stripe.

First claim

What is claimed is:

1. A non-transitory machine readable medium storing instructions, which when executed by a processing resource of a first node of a plurality of nodes of a cluster of a distributed storage management system, cause the first node to:
identify a failed Redundant Array of Independent Disks (RAID) stripe, wherein availability of data blocks is supported by way of a redundancy scheme implemented by the cluster;
identify, by a key-value (KV) store of the first node, a plurality of block identifiers (IDs) associated with the failed RAID stripe, wherein the KV store contains data blocks as values and their respective block IDs as keys; and
for each block identifier (ID) of the plurality of block IDs, restore a data block corresponding to the block ID by:
reading the data block from a node of the plurality of nodes having a redundant copy of the data block; and
writing the redundant copy of the data block to a storage area of the first node that is unaffected by the failed RAID stripe.

2. The non-transitory machine readable medium of claim 1 , wherein restoration of data blocks corresponding to the plurality of block IDs is performed responsive to a request by the KV store for performance of a RAID stripe resynchronization.

3. The non-transitory machine readable medium of claim 1 , wherein the redundancy scheme comprises replication of data blocks among multiple nodes of the plurality of nodes of the cluster.

4. The non-transitory machine readable medium of claim 3 , wherein the replication of data blocks is in accordance with a replication factor of 2 or more.

5. The non-transitory machine readable medium of claim 1 , wherein the failed RAID stripe is identified by a metadata service of the first node, wherein the metadata service maintains a mapping between logical block addresses to corresponding block IDs.

6. The non-transitory machine readable medium of claim 5 , wherein the failed RAID stripe is identified responsive to the metadata service encountering data block read errors for a number of block IDs within a particular RAID stripe meeting or exceeding a predetermined or configurable threshold.

7. The non-transitory machine readable medium of claim 1 , wherein a file system instance of the node includes a redundant block service to which a bin with which the block ID is associated has been assigned in addition to a block service of a file system instance of the first node.

8. A method comprising:
identifying, by a first node of a plurality of nodes of a cluster of a distributed storage management system, a failed Redundant Array of Independent Disks (RAID) stripe, wherein availability of data blocks is supported by way of a redundancy scheme implemented by the cluster;
identify, by a key-value (KV) store of the first node, a plurality of block identifiers (IDs) associated with the failed RAID stripe, wherein the KV store contains data blocks as values and their respective block IDs as keys; and
responsive to said identifying, for each block identifier (ID) of the plurality of block IDs, restoring a data block corresponding to the block ID by:
reading the data block from a node of the plurality of nodes having a redundant copy of the data block; and
writing the redundant copy of the data block to a storage area of the first node that is unaffected by the failed RAID stripe.

9. The method of claim 8 , wherein said restoring is performed responsive to a request by the KV store for performance of a RAID stripe resynchronization.

10. The method of claim 8 , wherein the redundancy scheme comprises replication of data blocks among multiple nodes of the plurality of nodes of the cluster.

11. The method of claim 8 , wherein the failed RAID stripe is identified by a metadata service of the first node, wherein the metadata service maintains a mapping between logical block addresses to corresponding block IDs.

12. A distributed storage management system comprising:
a processing resource; and
a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause a first node of a plurality of nodes of a cluster of the distributed storage management system to:
identify a failed Redundant Array of Independent Disks (RAID) stripe, wherein availability of data blocks is supported by way of a redundancy scheme implemented by the cluster;
identify, by a key-value (KV) store of the first node, a plurality of block identifiers (IDs) associated with the failed RAID stripe, wherein the KV store contains data blocks as values and their respective block IDs as keys; and
for each block identifier (ID) of the plurality of block IDs, restore a data block corresponding to the block ID by:
reading the data block from a node of the plurality of nodes having a redundant copy of the data block; and
writing the redundant copy of the data block to a storage area of the first node that is unaffected by the failed RAID stripe.

13. The distributed storage management system of claim 12 , wherein restoration of data blocks corresponding to the plurality of block IDs is performed responsive to a request by the KV store for performance of a RAID stripe resynchronization.

14. The distributed storage management system of claim 12 , wherein the redundancy scheme comprises replication of data blocks among multiple nodes of the plurality of nodes of the cluster.

15. The distributed storage management system of claim 12 , wherein the failed RAID stripe is identified by a metadata service of the first node, wherein the metadata service maintains a mapping between logical block addresses to corresponding block IDs.

16. The distributed storage management system of claim 15 , wherein the failed RAID stripe is identified responsive to the metadata service encountering data block read errors for a number of block IDs within a particular RAID stripe meeting or exceeding a predetermined or configurable threshold.

17. The distributed storage management system of claim 12 , wherein a file system instance of the node includes a redundant block service to which a bin with which the block ID is associated has been assigned in addition to a block service of a file system instance of the first node.

18. A non-transitory machine readable medium storing instructions, which when executed by a processing resource of a first node of a plurality of nodes of a cluster of a distributed storage management system, cause the first node to:
maintain a mapping between logical block addresses and corresponding block identifiers (IDs);
identify, by a metadata service of the first node, a failed Redundant Array of Independent Disks (RAID) stripe, wherein availability of data blocks is supported by way of a redundancy scheme implemented by the cluster; and
for each block identifier (ID) of a plurality of block IDs associated with the failed RAID stripe, restore a data block corresponding to the block ID by:
reading the data block from a node of the plurality of nodes having a redundant copy of the data block; and
writing the redundant copy of the data block to a storage area of the first node that is unaffected by the failed RAID stripe.

19. The non-transitory machine readable medium of claim 18 , wherein the instructions further cause a key-value (KV) store of the first node to identify the plurality of block IDs, wherein the KV store contains data blocks as values and their respective block IDs as keys.

20. The non-transitory machine readable medium of claim 1

Assignees

• Netapp Inc

Inventors

• Sun Wei
• Thoppil Anil Paul
• Vasu Anne Maria

Classifications

• G06F11/1662 Primary

  the resynchronized component or unit being a persistent storage device (re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008) · CPC title

• G06F3/0622

  in relation to access · CPC title

• G06F3/064

  Management of blocks · CPC title

• G06F3/0679

  Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP] · CPC title

• G06F11/1088

  Reconstruction on already foreseen single or plurality of spare disks · CPC title