Distributed file system that provides scalability and resiliency

Patent metadata

Field	Value
Publication number	US-12591552-B2
Application number	US-202418780353-A
Country	US
Kind code	B2
Filing date	Jul 22, 2024
Priority date	Jun 7, 2021
Publication date	Mar 31, 2026
Grant date	Mar 31, 2026

Abstract

A distributed storage management system comprising nodes that form a cluster, a distributed block layer that spans the nodes in the cluster, and file system instances deployed on the nodes. Each file system instance comprises a data management subsystem and a storage management subsystem disaggregated from the data management subsystem. The storage management subsystem comprises a node block store that forms a portion of the distributed block layer and a storage manager that manages a key-value store and virtualized storage supporting the node block store. A file system volume hosted by the data management subsystem maps to a logical block device hosted by the virtualized storage in the storage management subsystem. The key-value store includes, for a data block of the logical block device, a key that comprises a block identifier for the logical block device and a value that comprises the data block.

First claim

What is claimed is:

1 . A method comprising:
hosting a file system volume on a node of a plurality of nodes of a storage cluster; exposing the file system volume to a client of the storage cluster by an instance of a data management subsystem that provides file and data management functions; mapping file system data and file system metadata of the file system volume to a logical block address of a data block of a logical block device of an instance of a storage management subsystem that provides storage and block management functions, wherein the storage management subsystem is disaggregated from the data management subsystem; mapping the logical block address to a block identifier containing information indicative of a block store within the storage cluster and a location within the block store where the data block resides; storing the block identifier as a key and the data block as a corresponding value in a key-value store; and scaling either or both (i) a number of instances of the data management subsystem in accordance with demands of the client independently of a number of instances of the storage management subsystem and (ii) the number of instances of the storage management subsystem in accordance with storage demands on the storage cluster independently of the number of instances of the data management subsystem.

2 . The method of claim 1 , wherein the data management subsystem comprises a stateless entity in a form of an executable or a library.

3 . The method of claim 1 , wherein both the instance of the data management subsystem and the instance of the storage management subsystem are associated with a same file system instance deployed on the node.

4 . The method of claim 1 , wherein the instance of the data management subsystem is associated with a first file system instance deployed on the node and wherein the instance of the storage management subsystem is associated with a second file system instance deployed on a second node of the plurality of nodes.

5 . The method of claim 1 , wherein the instance of the data management subsystem runs on an application node on which the client resides to facilitate scaling of the data management subsystem along with the client.

6 . The method of claim 1 , wherein the logical block device comprises a logical unit number (LUN) device.

7 . The method of claim 1 , wherein the file system metadata and the file system data are stored in hash form in the logical block address.

8 . The method of claim 1 , wherein the block identifier comprises a content hash of the data block.

9 . A distributed storage system comprising:
a plurality of nodes that form a cluster, wherein the cluster includes one or more central processing unit (CPU) resources and instructions that when executed by the one or more CPU resources cause the distributed storage system to: host a file system volume on a node of the plurality of nodes; expose the file system volume to a client of the cluster by an instance of a data management subsystem that provides file and data management functions; map file system data and file system metadata of the file system volume to a logical block address of a data block of a logical block device of an instance of a storage management subsystem that provides storage and block management functions, wherein the storage management subsystem is disaggregated from the data management subsystem; map the logical block address to a block identifier containing information indicative of a block store within the cluster and a location within the block store where the data block resides; store the block identifier as a key and the data block as a corresponding value in a key-value store; and scale either or both (i) a number of instances of the data management subsystem in accordance with demands of the client independently of a number of instances of the storage management subsystem and (ii) the number of instances of the storage management subsystem in accordance with storage demands on the cluster independently of the number of instances of the data management subsystem.

10 . The distributed storage system of claim 9 , wherein the data management subsystem comprises a stateless entity in a form of an executable or a library.

11 . The distributed storage system of claim 9 , wherein both the instance of the data management subsystem and the instance of the storage management subsystem are associated with a same file system instance deployed on the node.

12 . The distributed storage system of claim 9 , wherein the instance of the data management subsystem is associated with a first file system instance deployed on the node and wherein the instance of the storage management subsystem is associated with a second file system instance deployed on a second node of the plurality of nodes.

13 . The distributed storage system of claim 9 , wherein the instance of the data management subsystem runs on an application node on which the client resides to facilitate scaling of the data management subsystem along with the client.

14 . A non-transitory machine readable medium storing instructions, which when executed by one or more central processing unit (CPU) resources of a distributed storage system, cause the distributed storage system to:
host a file system volume on a node of a plurality of nodes of a storage cluster; expose the file system volume to a client of the storage cluster by an instance of a data management subsystem that provides file and data management functions; map file system data and file system metadata of the file system volume to a logical block address of a data block of a logical block device of an instance of a storage management subsystem that provides storage and block management functions, wherein the storage management subsystem is disaggregated from the data management subsystem; map the logical block address to a block identifier containing information indicative of a block store within the storage cluster and a location within the block store where the data block resides; store the block identifier as a key and the data block as a corresponding value in a key-value store; and scale either or both (i) a number of instances of the data management subsystem in accordance with demands of the client independently of a number of instances of the storage management subsystem and (ii) the number of instances of the storage management subsystem in accordance with storage demands on the storage cluster independently of the number of instances of the data management subsystem.

15 . The non-transitory machine readable medium of claim 14 , wherein the data management subsystem comprises a stateless entity in a form of an executable or a library.

16 . The non-transitory machine readable medium of claim 14 , wherein both the instance of the data management subsystem and the instance of the storage management subsystem are associated with a same file system instance deployed on the node.

17 . The non-transitory machine readable medium of claim 14 , wherein the instance of the data management subsystem is associated with a first file system instance deployed on the node and wherein the instance of the storage management subsystem is associated with a second file system instance deployed on a second node of the plurality of nodes.

18 . The non-transitory machine readable medium of claim 14 , wherein the instance of the data management subsystem runs on an application node on which the client resides to facilitate scaling of the data management subsystem along with the cli

Assignees

• Netapp Inc

Inventors

• Dronamraju Ravikanth
• Subramanian Ananthan
• Mccarthy Daniel
• Cason Christopher
• Banerjee Arindam

Classifications

• G06F16/182

  Distributed file systems · CPC title

• G06F9/5077

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title

• G06F11/2023

  Failover techniques · CPC title

• G06F11/1662

  the resynchronized component or unit being a persistent storage device (re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008) · CPC title

• G06F11/0751

  Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06) · CPC title