System and method for adaptive data placement within a distributed file system

Patent metadata

Field	Value
Publication number	US-9710478-B2
Application number	US-201414268059-A
Country	US
Kind code	B2
Filing date	May 2, 2014
Priority date	May 2, 2014
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

A system and method for adaptive data placement in a distributed file system is provided. Upon creation of a new subdirectory, a first heuristic procedure is formed to determine whether the newly created subdirectory should be created locally or on a remote flexible volume. Should it be determination be made to create or store the subdirectory on a remote flexible volume, a second heuristic procedure determines which of a plurality of flexible volumes should store the newly created subdirectory.

First claim

What is claimed is:

1. A method comprising:
receiving a command to create a data container;
assigning, by a remote access module, the data container to one of a plurality of categories, wherein the plurality of categories comprises data containers being created off of a root directory, data containers being created off directory that is within a root directory and data containers being created off directory that is within a root directory;
calculating, by the remote access module, an urgency value for the data container;
calculating, by the remote access module, a disparity value for the data container;
calculating, by the remote access module, a local factor for the data container; determining, by the remote access module, whether the data container is to be stored remotely, according to one of the plurality of categories, and the local factor; and
in response to determining that the data container is to be stored remotely, determining, by the remote access module, one of a plurality of remote storage containers to store the data container by calculating a recent percentage for each of the plurality of remote storage containers, calculating a target percentage for each of the plurality of remote storage containers and selecting one of the plurality of remote storage containers having a greatest difference between the calculated recent percentage and the calculated target percentage.

2. The method of claim 1 wherein the data container comprises a subdirectory.

3. The method of claim 1 wherein the storage containers comprise flexible volumes.

4. The method of claim 1 wherein, in response to determining that the data container is not to be stored remotely, creating the data container on a storage container locally connected to a node.

5. The method of claim 1 wherein the local factor is calculated using the urgency value and the disparity value.

6. The method of claim 5 wherein the local factor is calculated by subtracting from a predefined value a value formed by multiplying the urgency value by the disparity value.

7. A network device comprising:
a memory containing machine readable medium comprising medium executable code having stored thereon instructions for performing a method of adaptive data placement of a data container;
a processing module coupled to the memory, the processor module configured to execute the machine executable code to:
receive a command to create a data container;
assign the data container to one of a plurality of categories, wherein the plurality of categories comprises data containers being created off of a root directory, data containers being created off directory that is within a root directory and data containers being created off directory that is within a root directory;
calculate an urgency value for the data container;
calculate a disparity value for the data container;
calculate a local factor for the data container;
determine whether the data container is to be stored remotely, according to one of the plurality of categories, and the local factor; and
in response to determining that the data container is to be stored remotely, determine, by the remote access module, one of a plurality of remote storage containers to store the data container by calculating a recent percentage for each of the plurality of remote storage containers, calculating a target percentage for each of the plurality of remote storage containers and selecting one of the plurality of remote storage containers having a greatest difference between the calculated recent percentage and the calculated target percentage.

8. The network device of claim 7 wherein the one or more remote storage containers comprise flexible volumes.

9. The network device of claim 7 wherein the data container comprises a subdirectory.

10. The network device of claim 7 wherein the processor module is further configured to execute the machine executable code to, in response to determining that the data container is not to be stored on the at least one storage containers stored on storage devices associated with the node, determine on which of the one or more remote storage containers the data container is to be created.

11. The network device of claim 10 wherein the processor module is further configured to execute the machine executable code to query each other node of the cluster via the network to obtain information to calculate the recent percentage value.

12. A non-transitory machine readable medium having stored thereon instructions for performing a method of adaptive data placement of a data container, comprising machine executable code which, when executed by at least one machine, causes the machine to:
receive a command to create a data container;
assign the data container to one of a plurality of categories, wherein the plurality of categories comprises data containers being created off of a root directory, data containers being created off directory that is within a root directory and data containers being created off directory that is within a root directory;
calculate an urgency value for the data container;
calculate a disparity value for the data container;
calculate a local factor for the data container;
determine whether the data container is to be stored remotely, according to one of the plurality of categories, and the local factor; and
in response to determining that the data container is to be stored remotely, determine one of a plurality of remote storage containers to store the data container by calculating a recent percentage for each of the plurality of remote storage containers, calculating a target percentage for each of the plurality of remote storage containers and selecting one of the plurality of remote storage containers having a greatest difference between the calculated recent percentage and the calculated target percentage.

13. The non-transitory machine readable medium of claim 12 wherein the one or more remote storage containers comprise flexible volumes.

Assignees

• Netapp Inc

Inventors

• Jernigan Iv Richard P
• Hyer Jr Robert Wyckoff
• Tennant Daniel

Classifications

• G06F16/9017

  using directory or table look-up (use of a directory or look-up table in file systems G06F16/13) · CPC title

• G06F16/16 Primary

  File or folder operations, e.g. details of user interfaces specifically adapted to file systems · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F16/134

  Distributed indices · CPC title