Dynamically scaling application and storage system functions based on a heterogeneous resource pool available for use by a distributed storage management system

Patent metadata

Field	Value
Publication number	US-12079242-B2
Application number	US-202218047774-A
Country	US
Kind code	B2
Filing date	Oct 19, 2022
Priority date	Oct 19, 2021
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

Systems and methods for scaling application and/or storage system functions of a distributed storage system based on a heterogeneous resource pool are provided. According to one embodiment, the distributed storage system has a composable, service-based architecture that provides scalability, resiliency, and load balancing. The distributed storage system includes a cluster of nodes each potentially having differing capabilities in terms of processing, memory, and/or storage. The distributed storage system takes advantage of different types of nodes by selectively instating appropriate services (e.g., file and volume services and/or block and storage management services) on the nodes based on their respective capabilities. Furthermore, disaggregation of these services, facilitated by interposing a frictionless layer (e.g., in the form of one or more globally accessible logical disks) therebetween, enables independent and on-demand scaling of either or both of application and storage system functions within the cluster while making use of the heterogeneous resource pool.

First claim

What is claimed is:

1. A non-transitory machine readable medium storing instructions, which when executed by a processing resource of a cluster including a plurality of nodes and collectively representing a distributed storage system cause the distributed storage system to:
add a new node to the cluster from a heterogeneous resource pool available for use by the cluster;
determine a resource capacity of the new node, including one or more of a storage resource capacity, a compute resource capacity, and a memory capacity associated with the new node;
based on the resource capacity of the new node, dynamically establish a configuration of the new node, wherein the configuration is indicative of whether one or both of a new storage management subsystem (SMS) and a new data management subsystem (DMS) are to be enabled on the new node, wherein responsive to the storage capacity being sufficient to accommodate block and storage management services, the new SMS is established within the configuration, and wherein responsive to the compute resource capacity being sufficient to accommodate file and volume services, the new DMS is established within the configuration; and
deploy, virtually, the new node in accordance with the configuration, wherein the new SMS and the new DMS are disaggregated from each other such that either the new SMS or the new DMS may operate independently within the new node or both the new SMS and the new DMS may operate within the new node.

2. The non-transitory machine readable medium of claim 1 , wherein storage capacity sufficient to accommodate block and storage management services comprises an amount of storage capacity sufficient to form a file system aggregate.

3. The non-transitory machine readable medium of claim 1 , wherein the instructions further cause the distributed storage system to initiate load balancing of responsibility for one or more of a set of one or more blocks, bins, or slices from an existing SMS of a node of the plurality of nodes to the new SMS of the new node.

4. The non-transitory machine readable medium of claim 1 , wherein the instructions further cause the distributed storage system to initiate load balancing of responsibility for one or more file system volumes from an existing DMS of a node of the plurality of nodes to the new DMS of the new node.

5. The non-transitory machine readable medium of claim 1 , wherein the new node is added to the heterogeneous resource pool in response to presence of a demand for an additional service provided by either the DMS or the SMS.

6. A method comprising:
adding, by a cluster of a plurality of nodes representing a distributed storage system, a new node to the cluster from a heterogeneous resource pool available for use by the cluster;
determining a resource capacity of the new node, including one or more of a storage resource capacity, a compute resource capacity, and a memory capacity associated with the new node;
based on the resource capacity of the new node, dynamically establishing a configuration of the new node, wherein the configuration is indicative of whether one or both of a new storage management subsystem (SMS) and a new data management subsystem (DMS) are to be enabled on the new node, wherein responsive to the storage capacity being sufficient to accommodate block and storage management services, the new SMS is established within the configuration, and wherein responsive to the compute resource capacity being sufficient to accommodate file and volume services, the new DMS is established within the configuration; and
deploying, virtually, the new node in accordance with the configuration, wherein the new SMS and the new DMS are disaggregated from each other such that either the new SMS or the new DMS may operate independently within the new node or both the new SMS and the new DMS may operate within the new node.

7. The method of claim 6 , wherein storage capacity sufficient to accommodate block and storage management services comprises an amount of storage capacity sufficient to form a file system aggregate.

8. The method of claim 6 , further comprising initiating load balancing of responsibility for one or more of a set of one or more blocks, bins, or slices from an existing SMS of a node of the plurality of nodes to the new SMS of the new node.

9. The method of claim 6 , further comprising initiating load balancing of responsibility for one or more file system volumes from an existing DMS of a node of the plurality of nodes to the new DMS of the new node.

10. The method of claim 6 , wherein the new node is added to the heterogeneous resource pool in response to presence of a demand for an additional service provided by either the DMS or the SMS.

11. A non-transitory machine readable medium storing instructions, which when executed by a processing resource of a cluster including a plurality of nodes and collectively representing a distributed storage system cause the distributed storage system to:
after identifying existence of a new node within a heterogeneous resource pool available for use by the cluster, add the new node to the cluster;
determine resource capacity associated with the new node;
responsive to the new node having sufficient storage capacity to accommodate block and storage management services, enable a storage management subsystem (SMS) within a configuration of the new node;
responsive to the new node having sufficient compute capacity to accommodate file and volume services, enable a data management subsystem (DMS) within the configuration; and
deploy, virtually, the new node in accordance with the configuration, wherein the SMS and the DMS are disaggregated from each other such that either the SMS or the DMS may operate independently within the new node or both the SMS and the DMS may operate within the new node, including:
provisioning the new node with a container including both the SMS and the DMS, wherein processes associated with one or more services associated with either or both of the SMS and the DMS are caused to be selectively launched based on the configuration; or
based on the configuration, selectively provisioning the new node with one or more of (i) a first set of one or more containers including the SMS and (ii) a second set of one or more containers including the DMS.

12. The non-transitory machine readable medium of claim 11 , wherein the new node comprises a new worker node of a container orchestration platform, a new VM, a new physical server, or a just a bunch of disks (JBOD) system.

13. The non-transitory machine readable medium of claim 11 , wherein the new node is added to the heterogeneous resource pool in response to presence of a demand for an additional service provided by either the DMS or the SMS.

14. The non-transitory machine readable medium of claim 11 , wherein the instructions further cause the distributed storage system to load balance a set of objects in a cluster database from one or more of the plurality of nodes to the new node.

15. The non-transitory machine readable medium of claim 14 , wherein the set of objects include a logical block device, a corresponding logical aggregate, and a corresponding file system volume.

16. A method comprising:
after identifying existence of a new node within a heterogeneous resource pool available for use by a cluster of a plurality of nodes representing a distributed storage system, adding the new node to the cluster;
determining resource capacity associated with the new node;
responsive to the new node having sufficient storage capacity to accommodate block and storage management services, ena

Assignees

• Netapp Inc

Inventors

• Banerjee Arindam
• Mccarthy Daniel
• Cason Christopher
• Subramanian Ananthan

Classifications

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title