Intelligent file system with transparent storage tiering

Patent metadata

Field	Value
Publication number	US-10042860-B2
Application number	US-201715595838-A
Country	US
Kind code	B2
Filing date	May 15, 2017
Priority date	Dec 15, 2014
Publication date	Aug 7, 2018
Grant date	Aug 7, 2018

Abstract

A file system manager implemented at a provider network identifies a storage device of a first group of storage devices of a provider network as an initial location of a file system object. Based on an access metric associated with the object, the file system manager initiates a transfer of contents of the object to a second storage device of a different storage device group, without receiving a client request specifying the transfer. In response to an access request received via a file system programmatic interface, contents of the object are provided from the second storage device. Based on a second access metric, the object is transferred back to the first group of storage devices.

First claim

What is claimed is:

1. A system, comprising:
a file system manager implemented at one or more computing devices; and
a plurality of groups of storage devices, including a first group of storage devices with a first capability level and a second group of storage devices with a different capability level;
wherein the file system manager is configured to:
determine an accessibility mode of a particular file system, wherein the accessibility mode is selected from a set which includes at least (a) a private mode and (b) a shared mode;
identify, based at least in part on the accessibility mode of the particular file system, at least one storage device of the first group as an initial location of a first file system object; and
initiate, based at least in part on a first metric associated with the first file system object, a first transfer of at least a portion of contents of the first file system object from the initial location to a second storage device of the second group, without receiving a corresponding request specifying that contents of the first file system object are to be transferred.

2. The system as recited in claim 1 , wherein the file system manager is configured to:
include, in a listing of file system objects of the particular file system provided subsequent to the first transfer, the first file system object, without providing an indication that the at least a portion of the first file system object has been transferred to the second storage device.

3. The system as recited in claim 1 , wherein the file system manager is configured to:
initiate, based at least in part on a second metric, a second transfer of the at least a portion of contents of the first file system object from the second group to the first group, without receiving a corresponding request specifying that contents of the first file system object are to be transferred.

4. The system as recited in claim 1 , wherein an initiation of the first transfer is based at least in part on a transfer policy indicated via a programmatic interface.

5. The system as recited in claim 4 , wherein the transfer policy includes an indication of an expected access pattern.

6. A method, comprising:
identifying, by a file system manager, at least a first storage device of a first group of storage devices as an initial location of a file system object, wherein the file system manager has access to a plurality of groups of storage devices with respective capability levels including the first group and a second group;
collecting one or more metrics associated with the file system object, including a first metric;
initiating, by the file system manager, a first transfer, based at least in part on the first metric, of at least a portion of contents of the file system object from the initial location to a second storage device of the second group of storage devices, without receiving a corresponding request specifying that contents of the file system object are to be transferred.

7. The method as recited in claim 6 , further comprising:
providing, by the file system manager from the second storage device subsequent to the first transfer, contents of the file system object in response to a request received via a programmatic interface.

8. The method as recited in claim 7 , further comprising:
initiating, by the file system manager, a second transfer, based at least in part on a second metric, of the at least a portion of contents of the file system object from the second group to the first group, without receiving a corresponding request specifying that contents of the file system object are to be transferred.

9. The method as recited in claim 6 , further comprising:
modifying, by the file system manager, one or more transfer policies associated with contents of the file system object, based at least in part on a machine learning-based analysis of one or more metrics.

10. The method as recited in claim 6 , wherein the file system object is part of a particular file system, further comprising:
determining, by the file system manager, an accessibility mode of the particular file system, wherein the accessibility mode is selected from a set which includes at least (a) a private mode enabling access to the particular file system from a single compute instance and (b) a shared mode enabling access to the particular file system from a plurality of compute instances;

wherein the identifying of the initial location of the first storage object is based at least in part on the accessibility mode of the particular file system.

11. The method as recited in claim 6 , wherein a particular group of the first and second groups of storage devices comprises: (a) locally-attached solid state drives (SSD) of instance hosts, (b) locally-attached rotating magnetic disk drives of instance hosts, (c) network-accessible solid state drives of a service implementing a block-level programmatic interface, (d) network-accessible magnetic disk drives of a service implementing a block-level programmatic interface, (e) devices of an object storage service implementing a web-services interface, (f) devices of a third-party storage service, or (g) storage devices attached to un-virtualized hosts.

12. The method as recited in claim 6 , wherein the initiating the first transfer is based at least in part on a transfer policy indicated programmatically.

13. The method as recited in claim 6 , further comprising:
identifying, by the file system manager, a second file system object whose contents are to be copied to the second storage device group from an initial location in the first storage device group;
initiating, by the file system manager, a copy of contents of the second file system object to the second storage device group, without deleting the contents from the first storage device group;
storing, by the file system manager, metadata indicating that the second file system object is a candidate for eviction from the first storage device group; and
in response to a subsequent access request directed to the second file system object,
determining, by the file system manager, that contents of the second file system object have not yet been evicted from the first storage device group; and
providing, from the first storage device group by the file system manager, contents of the second file system object.

14. The method as recited in claim 13 , further comprising:
determining, by the file system manager after providing the contents of the second file system object in response to the subsequent access request, that an amount of free space available at a particular storage device of the first storage device group has fallen below a threshold level; and
deleting the contents of the second file system object from the first storage device group.

15. The method as recited in claim 13 , wherein the metadata includes an indication that after the second file system is designated as a candidate for eviction from the first storage device group, billing for the presence of the second file system object in the first storage device group is not to be performed.

16. The method as recited in claim 6 , further comprising:
storing, by the file system manager, an encrypted version of the file system object at the first storage device after identifying at least the first storage device as the initial location of the file system object.

17. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors cause the one or more processors to:
identify at least a first storage device of a netw

Assignees

• Amazon Tech Inc

Inventors

• Krishnan Karthikeyan
• Parthasarathy Akshai
• Sait Abdul Sathar

Classifications

• G06F3/0647

  Migration mechanisms · CPC title

• G06F3/0611

  in relation to response time · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F16/185 Primary

  Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof (details of archiving G06F16/11) · CPC title

• G06F16/122

  using management policies (point-in-time backing up or restoration of persistent data G06F11/1446; file migration policies for HSM systems G06F16/185) · CPC title