Distributed file system gateway

Patent metadata

Field	Value
Publication number	US-9575974-B2
Application number	US-201314137706-A
Country	US
Kind code	B2
Filing date	Dec 20, 2013
Priority date	Oct 23, 2013
Publication date	Feb 21, 2017
Grant date	Feb 21, 2017

Abstract

Technology is disclosed for managing data in a distributed file system (“the technology”). The technology can gather metadata information associated with the data stored within a first file system, store the metadata information in association with a data identifier within a second file system, retrieve the stored metadata information using the data identifier from within the second file system and locate and retrieve the data associated with the metadata information from within first file system.

First claim

What is claimed is:

1. A method, comprising:
receiving a request, by a data node server, to access data through a first distributed file system, the request including a first data identifier associated with the data for accessing the data through the first distributed file system; wherein the received first data identifier is used by the data node server to determine that data associated with the first identifier has been evicted from the first distributed file system;
identifying, utilizing the received first data identifier, a second data identifier associated with the data identified as being evicted from the first file system, the second data identifier being stored within the first distributed file system and utilized for generating a request for accessing the data evicted from the first file system and stored within a second distributed file system;
retrieving, utilizing the identified second data identifier, the data from within the second distributed file system;
removing any indication that the data retrieved from the second distributed file system has been evicted from the first distributed file system;
converting the retrieved data from a first format of the second distributed file system to a second format of the first distributed file system;
storing, by a gateway manager in the data node server, the converted data within the first distributed file system, wherein the stored data is retrieved from within the first distributed file system in response to another request for the data; and
providing access to the converted retrieved data through the first distributed file system.

2. The method of claim 1 , further comprising:
receiving a request to store the data within the first file system, the request to store the data including the second data identifier associated with the data;
storing the second data identifier associated with the data within the first distributed file system; and
storing information indicative of the data being evicted from the first distributed file system.

3. The method of claim 1 , wherein the first distributed file system and the second distributed file system are different types of file systems.

4. The method of claim 3 , wherein the first distributed file system is a Hadoop Distributed File System (HDFS), wherein the second distributed file system is a Network File System (NFS).

5. The method of claim 4 , wherein the NFS stores the data in the first format, wherein the first format is a file level format and the HDFS stores the data in the second format, wherein the second format is a block level format.

6. The method of claim 5 , wherein the first data identifier includes a data block identifier (ID), wherein the second data identifier includes a file pathname.

7. The method of claim 1 , wherein the second data identifier includes metadata information that enables the first distributed file system to locate and retrieve the requested data from the second distributed file system.

8. The method of claim 5 , wherein the second data identifier stored in the first distributed file system includes one or more sub data identifiers, each of the sub data identifiers corresponding to a portion of the data stored within the second distributed file system.

9. A system, comprising:
a data node to receive a request to copy data from a first distributed file system to a second distributed file system;
a gateway client to gather metadata information associated with the data stored within the first distributed file system, the metadata information including information to locate and retrieve the requested data from within the first distributed file system; wherein the gateway client marks data identifiers of data blocks as being evicted when metadata information is stored within the first distributed file system but data associated with the data identifiers is stored at the second distributed file system;
a gateway manager to:
store the gathered metadata information in association with a data identifier within the second distributed file system, the data identifier being used to request access to the data via the second distributed file system;

a chunk store manager to convert the retrieved data from a first format to a second format;
the gateway manager further to store the converted data within the second file system, wherein the stored data can be retrieved from within the second distributed file system in response to another request for the data; wherein the gateway manager in response to the another request determines that data identifier in the other request is marked as being evicted from the first distributed file system, retrieves metadata information from the first distributed file system; and generates a request for the data using the retrieved metadata information to request the data from the second distributed file system; and
the data node further to send a confirmation indicating a completion of the data copy request; wherein the data node stores.

10. The system of claim 9 , further comprising:
the data node to receive a request to access the data through the second distributed file system, the request including the data identifier;
the gateway manager to:
utilize the received data identifier to gather the metadata information, associated with the requested data, from within the second distributed file system, and
provide access to the retrieved data through the second distributed file system; and

the gateway client to utilize the gathered metadata information to retrieve the data from within the first distributed file system.

11. The system of claim 9 , wherein the first distributed file system and the second distributed file system are different types of file systems.

12. The system of claim 11 , wherein the first distributed file system is a Network File System (NFS), wherein the second distributed file system is a Hadoop Distributed File System (HDFS).

13. The system of claim 12 , wherein the data identifier includes a data block identifier (ID), wherein the metadata information include a file pathname.

14. A non-transitory computer readable storage medium storing computer executable instructions, comprising:
instructions for receiving a request to access data through a first distributed file system, the request including a first data identifier associated with the data for accessing the data through the first distributed file system; wherein the received first data identifier is used by the data node server to determine that data associated with the first identifier has been evicted from the first distributed file system;
instructions for identifying, utilizing the received first data identifier, a second data identifier associated with the data identified as being evicted from the first file system, the second data identifier being stored within the first distributed file system and utilized for generating a request for accessing the data evicted from the first file system and stored within a second distributed file system;
instructions for retrieving, utilizing the identified second data identifier, the data from within the second distributed file system;
instructions for removing any indication that the data retrieved from the second distributed file system has been evicted from the first distributed file system;
instructions for converting the retrieved data from a first format of the second distributed file system to a second format of the first distributed file system;
instructions for storing the converted data within the first distributed file system, wherein the stored data can be retrieved from within the first distributed file system in response to anothe

Assignees

• Netapp Inc

Inventors

• Muthyala Kartheek
• Makkar Gaurav
• Suresh Arun
• Narayanamurthy Srinivasan

Classifications

• G06F16/116 Primary

  Details of conversion of file system types or formats · CPC title

• G06F16/1865

  Transactional file systems · CPC title

• G06F16/18

  File system types · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F16/134

  Distributed indices · CPC title