Partitioning a distributed filesystem namespace

Patent metadata

Field	Value
Publication number	US-10983953-B2
Application number	US-201916265412-A
Country	US
Kind code	B2
Filing date	Feb 1, 2019
Priority date	Feb 1, 2019
Publication date	Apr 20, 2021
Grant date	Apr 20, 2021

Abstract

The described technology is generally directed towards domains that data services can use to collect files of a global filesystem namespace into groups. A data service (a domain patron) creates a domain, e.g., for a particular directory, and a domain manager associates files under that directory with domain membership information. Thereafter, the data service can use the domain membership information associated with a file to determine whether to include that file in a data service operation. In one implementation the membership information is maintained in file metadata, facilitating fast and efficient retrieval of the information in near constant time. Exclusion from a domain, hard links, renames and alternate data streams are among the various aspects supported by the domain technology.

First claim

What is claimed is:

1. A method, comprising:
receiving, by a system comprising a processor, a first request to create a domain on a directory of a filesystem;
in response to the first request,
creating the domain on the directory,
associating the directory with the domain, and
in response to the directory not being empty, associating any nodes under the directory with the domain;

receiving a second request to read from a binary large object of data stored in the domain on the directory;
in response to the second request, performing an iterative read of the binary large object of data, wherein a lock is held for a duration of the iterative read to prevent the binary large object of data from changing during the iterative read;
receiving a third request with respect to a file object stored in one of the nodes under the directory; and
in response to the third request,
determining whether domain membership information associated with the file object is up to date,
in response to the domain membership information of the file object being determined to be up to date, returning the domain membership information, and
in response to the domain membership information of the file object being determined to be out of date, updating the domain membership information by obtaining updated domain membership information from a parent node of the file object, and returning the updated domain membership, wherein the domain membership information is cached in a domain cache, and wherein invalidation of the domain cache is performed using a global lock of each domain being invalidated.

2. The method of claim 1 , further comprising, in further response to the first request, storing an identifier of the domain in a domain data structure.

3. The method of claim 1 , further comprising, in further response to the first request, updating a generation value in a domain data structure.

4. The method of claim 1 , wherein the directory is not empty, and wherein the associating the any of the nodes under the directory with the domain comprises tagging respective any attributes associated with the any of the nodes with an identifier of the domain to designate that the any of the nodes are members of the domain.

5. The method of claim 1 , wherein the directory is not empty, wherein the node comprises the file object, and wherein the associating the any of the nodes under the directory with the domain comprises traversing a directory hierarchy associated with the directory, and tagging a file attribute associated with the file object with an identifier of the domain to designate that the any of the nodes are members of the domain when the file object is reached during the traversing.

6. The method of claim 1 , wherein the directory is an empty directory, and further comprising, detecting a new node added to the directory, and in response to the detecting, associating the new node with the domain.

7. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
in response to a first request to create a domain on a directory of a filesystem,
creating the domain on the directory,
associating the directory with the domain, and
in response to the directory not being empty, associating any nodes under the directory with the domain;

receiving a second request to read from a binary large object of data stored in the domain on the directory;
in response to the second request, performing an iterative read of the binary large object of data, wherein a lock is held for a duration of the iterative read to prevent the binary large object of data from changing during the iterative read;
receiving a third request with respect to a file object stored in one of the nodes under the directory; and
in response to the third request, determining whether domain membership information associated with the file object is up to date,
in response to the domain membership information of the file object being determined to be up to date, returning the domain membership information, and
in response to the domain membership information of the file object being determined to be out of date, updating the domain membership information, the updating comprising obtaining updated domain membership information associated with a parent node of the file object, and returning the updated domain membership, wherein the domain membership information is cached in a domain cache, and wherein invalidation of the domain cache is performed using a global lock of each domain being invalidated.

8. The system of claim 7 , wherein the associating the directory with the domain comprises associating the directory with the domain membership information, and associating the file object with the domain membership information.

9. The system of claim 7 , wherein the operations further comprise, maintaining the domain membership information in association with the file object comprising maintaining a domain identifier of the domain in metadata of the file object.

10. The system of claim 7 , wherein the operations further comprise, detecting an operation that renames the directory above the file object to a namespace corresponding to a second domain, obtaining first domain membership data of the first domain and second domain membership data of the second domain, and, in response to the first domain membership data being determined to be different from the second membership data, updating the domain membership information of the file object based on the second membership data.

11. The system of claim 7 , wherein the directory is a first directory, wherein the domain is a first domain, and wherein the operations further comprise, detecting an operation that provides a hard link from the first directory to a second directory corresponding to a second domain, and updating the domain membership information associated with the file object based on domain data associated with the second domain.

12. The system of claim 11 , wherein the operations further comprise, detecting an operation that removes the hard link between the first directory and the second directory, and updating the domain membership information associated with the file object based on the file object being determined to be no longer hard linked via the first directory to the second directory.

13. The system of claim 7 , wherein the domain membership information of the file object comprises a first domain identifier and a second domain identifier, and wherein the operations further comprise, adding a third domain identifier to the domain membership information, in which the third domain identifier logically excludes the first domain identifier from the domain membership information.

14. The system of claim 13 , wherein the returning the domain membership information comprises generating a response by excluding the first identifier from the response based on the third identifier, and returning the response comprising the second identifier.

15. The system of claim 7 , wherein the domain membership information of the file object comprises a first domain identifier, and wherein the operations further comprise, maintaining the first domain identifier in a domain data structure in conjunction with data service-specific data.

16. The system of claim 7 , wherein the determining whether the domain membership information of the file object is up to date comprises evaluating a domain membership information generation number associated with the domain membership information of the file object agains

Assignees

• Emc Ip Holding Co Llc

Inventors

• Rajawat Mayank
• Naujok Barry
• Fahadullah Mohd

Classifications

• G06F16/182 Primary

  Distributed file systems · CPC title

• G06F16/185

  Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof (details of archiving G06F16/11) · CPC title

• H04L67/06

  specially adapted for file transfer, e.g. file transfer protocol [FTP] · CPC title

• G06F16/13 Primary

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title