Persistent cache layer to tier data to cloud storage

Patent metadata

Field	Value
Publication number	US-11061868-B1
Application number	US-201715581370-A
Country	US
Kind code	B1
Filing date	Apr 28, 2017
Priority date	Apr 28, 2017
Publication date	Jul 13, 2021
Grant date	Jul 13, 2021

Abstract

Implementations are provided herein for having at least two data streams associated with each file in a file system. The first, a cache overlay layer, can store additional state information on a per block basis that details whether each individual block of file data within the cache overlay layer is clean, dirty, or indicates that a write back to the storage layer is in progress. The second, a storage layer, can be a use case defined repository that can transform data using data augmentation methods as a part of tiering data to external repositories. File system operations directed to the cache overlay layer can be processed asynchronously from file system operations directed to the storage layer.

First claim

What is claimed is:

1. A method comprising:
establishing at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establishing a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a set of cloud storage metadata; and
processing file system operations received from a file system client using the cache overlay layer asynchronously from processing file system operations directed to the storage layer, wherein processing file system operations includes:
processing a read operation based on a state of the cache overlay layer, and
processing a write operation by writing the write operation to the cache overlay layer and marking the chunk state for each chunk of file data implicated by the write operation.

2. The method of claim 1 , wherein the set of cloud storage metadata includes at least one of a cloud metadata object and a cloud data object.

3. The method of claim 1 , wherein processing file system operations received from the file system client includes processing a read operation targeted to a file, wherein processing the read operation includes:
determining a set of cloud data objects associated with the file based on the set of cloud storage metadata associated with the storage layer inode of the file;
retrieving a subset of the set of cloud data objects from a cloud storage location based on the set of cloud storage metadata; and
filling the subset of the set of cloud data objects into the cache overlay.

4. The method of claim 3 , further comprising:
in response to retrieving the subset of the set of cloud data objects, at least one of decompressing or decrypting the subset of the set of cloud objects.

5. The method of claim 1 , wherein processing file system operations includes processing a write-back operation, wherein processing the write-back operation includes:
identifying a set of chunks in the cache overlay inode that have chunk states marked as dirty; and
changing the chunk state of the set of chunks to write-back-in-progress.

6. The method of claim 1 , wherein processing file system operations includes processing a convert-and-store-metadata operation for a file, wherein processing the convert-and-store-metadata operation includes:
identifying a set of chunks in the cache overlay inode of the file that have chunk states marked as write-back-in-progress;
generating a modified set of cloud storage metadata based on the identified set of chunks;
associating the modified set of cloud storage metadata with the storage layer inode;
storing the set of chunks and the modified set of cloud storage metadata in a cloud storage location based on the modified set of cloud storage metadata; and
changing the chunk state of the set of chunks to clean.

7. The method of claim 6 , further comprising:
in response to identifying the set of chunks, at least one of encrypting or compressing the set of chunks based on the set of cloud storage metadata, wherein generating the modified set of cloud storage metadata is further based on the at least one encrypting or compressing.

8. A system comprising at least one storage device and at least one hardware processor configured to:
establish at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establish a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a set of cloud storage metadata, wherein the cloud storage metadata includes at least one or more of a reference to a cloud metadata; and
process file system operations received from a file system client using the cache overlay layer asynchronously from processing file system operations directed to the storage layer, wherein processing file system operations includes:
processing a read operation based on a state of the cache overlay layer, and
processing a write operation by writing the write operation to the cache overlay layer and marking the chunk state for each chunk of file data implicated by the write operation.

9. The system of claim 8 , wherein the set of cloud storage metadata includes at least one of a cloud metadata object and a cloud data object.

10. The system of claim 8 , further configured to:
process a read operation from the file system client targeted to a file, wherein processing the read operation includes:
determining a set of cloud data objects associated with the file based on the set of cloud storage metadata associated with the storage layer inode of the file;
retrieving a subset of the set of cloud data objects from a cloud storage location based on the set of cloud storage metadata; and
filling the subset of the set of cloud data objects into the cache overlay.

11. The system of claim 10 , further configured to:
in response to retrieving the subset of the set of cloud data objects, at least one of decompressing or decrypting the subset of the set of cloud objects.

12. The system of claim 8 , further configured to:
process a write-back operation, wherein processing the write-back operation includes:
identifying a set of chunks in the cache overlay inode that have chunk states marked as dirty; and
changing the chunk state of the set of chunks to write-back-in-progress.

13. The system of claim 8 , further configured to:
Process a convert-and-store-metadata operation for a file, wherein processing the convert-and-store-metadata operation includes:
identifying a set of chunks in the cache overlay inode of the file that have chunk states marked as write-back-in-progress;
generating a modified set of cloud storage metadata based on the identified set of chunks;
associating the modified set of cloud storage metadata with the storage layer inode;
storing the set of chunks and the modified set of cloud storage metadata in a cloud storage location based on the modified set of cloud storage metadata; and
changing the chunk state of the set of chunks to clean.

14. The system of claim 13 , further configured to:
in response to identifying the set of chunks, at least one of encrypt or compress the set of chunks based on the set of cloud storage metadata, wherein generating the modified set of cloud storage metadata is further based on the at least one encrypting or compressing.

15. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
establishing at least two data streams for each file in a file system, wherein a first data stream is associated with a cache overlay layer and a second data stream is associated with a storage layer;
establishing a logical inode tree that at least maps each file in the file system to a cache overlay layer inode and a storage layer inode, wherein the cache overlay layer inode contains metadata identifying a chunk state for each chunk of file data, and wherein the storage layer inode is associated with a set of cloud storage metadata; and
processing file system operations received from a file system client using the cache overlay layer async

Assignees

• EMC IP Holding Company
• Emc Ip Holding Co Llc

Inventors

• Popovich Evgeny
• Laier Max
• Cande Edward

Classifications

• H04L67/568

  Storing data temporarily at an intermediate stage, e.g. caching · CPC title

• G06F12/0804

  with main memory updating (G06F12/0806 takes precedence) · CPC title

• G06F2212/1024

  Latency reduction · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F16/1767

  Concurrency control, e.g. optimistic or pessimistic approaches · CPC title