Resource efficient scale-out file systems

Patent metadata

Field	Value
Publication number	US-9916258-B2
Application number	US-201113077779-A
Country	US
Kind code	B2
Filing date	Mar 31, 2011
Priority date	Mar 31, 2011
Publication date	Mar 13, 2018
Grant date	Mar 13, 2018

Abstract

According to one embodiment, a file system (FS) of a storage system is partitioned into a plurality of FS partitions, where each FS partition stores segments of data files. In response to an input and output (IO) request for accessing a first of the FS partitions, a second of the FS partitions is selected that is currently in a ready state for access. The second FS partition is then removed from the ready state for access. The first FS partition is brought into a ready state for access.

First claim

What is claimed is:

1. A computer-implemented method for providing a file system, the method comprising:
partitioning a file system (FS) of a storage system into a plurality of FS partitions by a FS partition manager executed by a processor of the storage system, each FS partition storing a plurality of files, wherein the plurality of FS partitions form a single logical file system using a single name space, wherein each of the FS partitions stores files having similar access patterns;
receiving an input and output (IO) request for accessing a first of the FS partitions, the first FS partition providing access to a first set of files;
in response to the IO request, selecting by the FS partition manager a second of the FS partitions that is currently in a ready state for access and that has no pending IO request for the second FS partition, the second FS partition providing access to a second set of files;
removing by the FS partition manager the second FS partition from the ready state for access, including unloading second FS metadata associated with the second FS partition from a system memory of the storage system, wherein the second FS metadata includes information for accessing the second set of files;
bringing the first FS partition into a ready state for access, including loading first FS metadata associated with the first FS partition into the system memory for servicing the IO request, wherein the first FS metadata includes information for accessing the first set of files, including
enqueuing the request in a pending IO queue of the file system if the first FS partition is not in the ready state for access,
scheduling an execution of the IO request if the first FS partition is in the ready state for access, and
incrementing a pending IO count associated with the first FS partition to indicate a number of IO requests currently pending for execution in the first FS partition; and

providing access to the first set of files via the first FS partition of the storage system and via the first FS metadata associated with the first FS partition loaded into the system memory, while the second FS metadata associated with the second FS partition is unloaded from the system memory, such that a memory space of the system memory required to store FS metadata is less than the system memory required to concurrently store both the first and the second FS metadata, wherein the first FS metadata includes information for accessing the first set of files, wherein the second FS metadata includes information for accessing the second set of files.

2. The method of claim 1 , wherein the first FS metadata includes name and directory information for accessing the files associated with the first FS partition, and wherein the second FS metadata includes name and directory information for accessing the files associated with the second FS partition.

3. The method of claim 1 , wherein removing the second FS partition from the ready state for access includes powering down a storage device associated with the second FS partition, and wherein bringing the first FS partition into the ready state for access includes powering up a storage device associated with the first FS partition.

4. The method of claim 1 , wherein the files stored in each FS partition are deduplicated within the FS partition.

5. The method of claim 1 , further comprising decrementing the pending IO count in response to a signal indicating that execution of the IO request has been completed.

6. The method of claim 1 , further comprising:
assigning a token having an initial value to the first FS partition when the first FS partition is being loaded; and
decrementing the value of the token associated with the first FS partition in response to the signal indicating a complete execution of the IO request.

7. The method of claim 6 , further comprising periodically decrementing the value of the token associated with the first FS partition.

8. The method of claim 7 , further comprising removing the first FS partition from the ready state for access if at least one of the pending IO count and the value of the token reaches a predetermined value.

9. The method of claim 1 , wherein the second FS partition is removed from the ready state for access only if there is no IO request pending for the second FS partition and the second FS partition has been in the ready state for access for a predetermined period of time.

10. The method of claim 1 , wherein files with similar access patterns include files with similar last modified times such that FS partitions of the storage system are accessed sequential so relevant files from one partition is accessed before moving on to the next partition.

11. The method of claim 1 , further comprising writing a file to a FS partition in a ready state, wherein the FS partition is selected based on a likely future access pattern of the file.

12. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform operations for managing resources of a file system, the operations comprising:
partitioning a file system (FS) of a storage system into a plurality of FS partitions by a FS partition manager executed by a processor of the storage system, each FS partition storing a plurality of files, wherein the plurality of FS partitions form a single logical file system using a single name space, wherein each of the FS partitions stores files having similar access patterns;
receiving an input and output (IO) request for accessing a first of the FS partitions, the first FS partition providing access to a first set of files;
in response to the IO request, selecting a second of the FS partitions that is currently in a ready state for access and that has no pending IO request for the second FS partition, the second FS partition providing access to a second set of files;
removing the second FS partition from the ready state for access, including unloading second FS metadata associated with the second FS partition from a system memory of the storage system, wherein the second FS metadata includes information for accessing the second set of files;
bringing the first FS partition into a ready state for access, including loading first FS metadata associated with the first FS partition into the system memory for servicing the IO request, wherein the first FS metadata includes information for accessing the first set of files, including
enqueuing the request in a pending IO queue of the file system if the first FS partition is not in the ready state for access,
scheduling an execution of the IO request if the first FS partition is in the ready state for access, and
incrementing a pending IO count associated with the first FS partition to indicate a number of IO requests currently pending for execution in the first FS partition; and

providing access to the first set of files via the first FS partition of the storage system and via the first FS metadata associated with the first FS partition loaded into the system memory, while the second FS metadata associated with the second FS partition is unloaded from the system memory, such that a memory space of the system memory required to store FS metadata is less than the system memory required to concurrently store both the first and the second FS metadata, wherein the first FS metadata includes information for accessing the first set of files, wherein the second FS metadata includes information for accessing the second set of files.

13. The computer-readable storage medium of claim 12 , wherein the first FS metadata includes name and directory information for accessing the fil

Assignees

• Mitra Soumyadeb
• Hsu Windsor W
• Emc Ip Holding Co Llc

Inventors

• Mitra Soumyadeb
• Hsu Windsor W

Classifications

• G06F9/5022 Primary

  Mechanisms to release resources · CPC title

• G06F12/12 Primary

  Replacement control · CPC title

• G06F12/0846

  Cache with multiple tag or data arrays being simultaneously accessible · CPC title

• G06F9/5016

  the resource being the memory · CPC title

• G06F16/17

  Details of further file system functions · CPC title