Filesystem hierarchical aggregate metrics

Patent metadata

Field	Value
Publication number	US-10459892-B2
Application number	US-201514595043-A
Country	US
Kind code	B2
Filing date	Jan 12, 2015
Priority date	Apr 23, 2014
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

A software and hardware facility persistently maintains metrics on directories at different levels within a tree of a filesystem. The facility hierarchically aggregates attributes of files contained by directories and stores them as metric values in each directory within a tree. The stored values represent summed or otherwise aggregated data from the descendant directories and files. The metric values represent aggregated attributes such as total space consumed by a directory and all the descendant directories and files, total number of files within a directory, total data blocks used by a directory and its descendant directories and data files, etc.

First claim

We claim:

1. A method for managing metrics for a filesystem over a network, wherein one or more processors execute instructions to perform actions, comprising:
receiving one or more changes affecting one or more files contained by a directory in a hierarchical tree of filesystem objects, wherein each filesystem object includes one directory or one file and corresponding inode data related to filesystem administration information, and wherein one or more attributes of a file are stored in that directory that contains the file itself;
storing one or more metrics at a directory on a direct path between a root directory and the directory that contains the one or more files affected by the change, wherein the one or more metrics are stored in each directory's associated inode data and each metric includes a value representing a summary of the one or more attributes of each file contained by one or more descendant directories of a subtree in the hierarchical tree, and wherein the subtree is based on the direct path between the directory and the root directory;
updating the values representing the summary for the one or more metrics stored in each directory's inode data on the direct path to reflect one or more alterations to one or more attributes of the one or more files contained by the directories of the subtree resulting from the change; and
sampling for at least one of the one or more attributes of each file stored in each directory of the hierarchical tree, wherein the sampling is weighted based on the one or more metrics stored in each directory's inode data that is associated with each sampled file attribute.

2. The method of claim 1 , further comprising:
maintaining a reconciled value and an unreconciled value associated with the one or more attributes of the one or more files contained by the one or more directories of the subtree resulting from the change, wherein the reconciled value corresponds to a value of an attribute reflected in a parent directory of each filesystem object corresponding to each directory of the subtree and the unreconciled value corresponds to a current value of the attribute reflected in each of these filesystem objects that correspond to directories of the subtree; and
updating the unreconciled value for the attribute to reflect the one or more alterations to that attribute in the corresponding filesystem object.

3. The method of claim 1 , further comprising:
in response to each update to the one or more directories' file system objects on the direct path, updating an unreconciled value for an attribute to equal a reconciled value in the filesystem object.

4. The method of claim 1 , wherein each metric value for the one or more files included in each filesystem object are updated asynchronously.

5. The method of claim 1 , wherein each metric value of the directory is updated asynchronously with respect to the received change to the one or more files contained in the directory.

6. The method of claim 1 , wherein a currency of each metric value is determined by checking whether a set of filesystem objects that includes the directory or each descendant directory on the direct path requires updating based on the change to the one or more files.

7. The method of claim 1 , wherein each metric value of a directory includes any one or more of a total directory count under a directory, a total file count under a directory, a total number of write operations on files contained in a directory, a moving sum of write operations on files contained in a directory, a total number of blocks used for file data, a moving sum of blocks used for file data, calendar time (ctime), and access time (atime).

8. A system for managing metrics for a filesystem over a network, comprising:
a memory that stores instructions;
a transceiver to manage communication over the network; and
one or more processors that execute the instructions to perform actions, comprising:
receiving one or more changes affecting one or more files contained by a directory in a hierarchical tree of filesystem objects, wherein each filesystem object includes one directory or one file and corresponding Mode data related to filesystem administration information, and wherein one or more attributes of a file are stored in that directory that contains the file itself;
storing one or more metrics at a directory on a direct path between a root directory and the directory that contains the one or more files affected by the change, wherein the one or more metrics are stored in each directory's associated inode data and each metric includes a value representing a summary of the one or more attributes of each file contained by one or more descendant directories of a subtree in the hierarchical tree, and wherein the subtree is based on the direct path between the directory and the root directory;
updating the values representing the summary for the one or more metrics stored in each directory's inode data on the direct path to reflect one or more alterations to one or more attributes of the one or more files contained by the directories of the subtree resulting from the change; and
sampling for at least one of the one or more attributes of each file stored in each directory of the hierarchical tree, wherein the sampling is weighted based on the one or more metrics stored in each directory's inode data that is associated with each sampled file attribute.

9. The system of claim 8 , further comprising:
maintaining a reconciled value and an unreconciled value associated with the one or more attributes of the one or more files contained by the one or more directories of the subtree resulting from the change, wherein the reconciled value corresponds to a value of an attribute reflected in a parent directory of each filesystem object corresponding to each directory of the subtree and the unreconciled value corresponds to a current value of the attribute reflected in each of these filesystem objects that correspond to directories of the subtree; and
updating the unreconciled value for the attribute to reflect the one or more alterations to that attribute in the corresponding filesystem object.

10. The system of claim 8 , further comprising:
in response to each update to the one or more directories' file system objects on the direct path, updating an unreconciled value for an attribute to equal a reconciled value in the filesystem object.

11. The system of claim 8 , wherein each metric value for the one or more files included in each filesystem object are updated asynchronously.

12. The system of claim 8 , wherein each metric value of the directory is updated asynchronously with respect to the received change to the one or more files contained in the directory.

13. The system of claim 8 , wherein a currency of each metric value is determined by checking whether a set of filesystem objects that includes the directory or each descendant directory on the direct path requires updating based on the change to the one or more files.

14. The system of claim 8 , wherein each metric value of a directory includes any one or more of a total directory count under a directory, a total file count under a directory, a total number of write operations on files contained in a directory, a moving sum of write operations on files contained in a directory, a total number of blocks used for file data, a moving sum of blocks used for file data, calendar time (ctime), and access time (atime).

15. A computer-readable non-transitory storage medium that includes a plurality of instructions, wherein execution of the instructions by one or more processors performs actions, comprising:

Assignees

• Qumulo Inc

Inventors

• Godman Peter
• O'Neill Kevin
• Urban Thomas
• Husted Justin
• Williams Graeme

Classifications

• G06F16/185 Primary

  Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof (details of archiving G06F16/11) · CPC title

• G06F9/467

  Transactional memory (G06F9/528 takes precedence) · CPC title

• G06F16/2358

  Change logging, detection, and notification (replication G06F16/27) · CPC title