Proactive optimizations at multi-tier file systems

Patent metadata

Field	Value
Publication number	US-10127234-B1
Application number	US-201514672086-A
Country	US
Kind code	B1
Filing date	Mar 27, 2015
Priority date	Mar 27, 2015
Publication date	Nov 13, 2018
Grant date	Nov 13, 2018

Abstract

A recommendations manager (RM) of a file system service identifies a file system object group (FSOG) for which proactive placement recommendations are to be generated using statistical analyzes. Usage metrics collected from the FSOG are used to train a model. Predictions obtained from the model are used to generate a recommendation to transfer a file system object proactively from one storage device group of the service to another. The recommendations are provided to an object migrator of the service to initiate the transfer.

First claim

What is claimed is:

1. A system, comprising:
a recommendations manager (RM) of a file system service, implemented at one or more computing devices of a provider network;
one or more object migrators of the file system service; and
a plurality of storage device groups (SDG) of the provider network, including a first SDG with a first storage performance capability level and a second SDG with a different storage performance capability level;
wherein the RM is configured to:
identify one or more file system object groups (FSOGs) for which respective sets of proactive placement recommendations (PPRs) are to be generated, including a first FSOG associated with a first client, and a second FSOG associated with a second client;
determine a first constraint on resources of the provider network to be used to generate a first set of PPRs for the first FSOG, and a second constraint on resources to be used to generate a second set of PPRs for the second FSOG;
obtain a first collection of usage metrics of the first FSOG, and a second collection of usage metrics of the second FSOG;
perform, (a) a first statistical analysis using the first collection of usage metrics and resources selected in accordance with the first constraint, and (b) a second statistical analysis using the second collection of usage metrics and resources selected in accordance with the first constraint;
generate, based at least in part on one or more predictions obtained from the first statistical analysis, the first set of PPRs, including at least a first recommendation to transfer a first file system object of the first FSOG from a particular storage device of the first SDG to a second storage device of the second SDG;
generate, based at least in part on one or more predictions obtained from the second statistical analysis, the second set of PPRs, including at least a second recommendation to transfer a second file system object of the second FSOG from a particular storage device of the second SDG to a different storage device of the first SDG; and
wherein the one or more object migrators are configured to:
initiate respective transfers of the first and second file system objects in accordance with the first and second recommendations.

2. The system as recited in claim 1 , wherein the first constraint is based at least in part on one or more of: (a) an identity of a client on whose behalf the first FSOG is created, (b) a size of the first FSOG, (c) an analysis of billing records associated with the first FSOG, (d) an available computing capacity at a machine learning service implemented at a provider network.

3. The system as recited in claim 1 , wherein the first statistical analysis comprises an implementation of a first modeling methodology selected from a methodology set comprising: (a) regression, (b) classification or (c) time series modeling, and wherein the second statistical analysis comprises an implementation of a different modeling methodology of the methodology set.

4. The system as recited in claim 1 , wherein the RM is further configured to:
classify, using a clustering model, users of the file system service into a plurality of categories based at least in part on respective file access patterns of the users, wherein the first set of PRRs is based at least in part on a result of the clustering model.

5. The system as recited in claim 1 , wherein to identify the first FSOG, the RM is configured to:
receive, via a programmatic interface of the file system service, an indication from a client of one or more file system objects for which the client has opted in for proactive placement recommendations.

6. A method, comprising:
performing, by a recommendations manager (RM) of a file system service whose data is distributed among a plurality of storage device groups (SDGs):
identifying one or more file system object groups (FSOGs) for which respective sets of proactive placement recommendations (PPRs) are to be generated using one or more machine learning models, including a first FSOG and a second FSOG;
training, (a) a first machine learning model using a first collection of usage metrics of the first FSOG, and (b) a second machine learning model using a second collection of usage metrics of the second FSOG; and
generating, based at least in part on one or more predictions obtained from the first machine learning model, a first set of PPRs for the first FSOG, including at least one recommendation to transfer a particular file system object of the first FSOG from its current SDG to a different SDG; and
generating, based at least in part on one or more predictions obtained from the second machine learning model, a second set of PPRs for the second FSOG, including at least one recommendation to transfer a particular file system object of the second FSOG from its current SDG to a different SDG.

7. The method as recited in claim 6 , further comprising:
determining, by the RM, a first constraint on machine learning resources of a provider network to be used to generate the first set of PPRs, wherein the training of the first machine learning model is performed using resources selected in accordance with the first constraint.

8. The method as recited in claim 7 , wherein the first constraint is based at least in part on one or more of: (a) an identity of a client on whose behalf the first FSOG is created, (b) a size of the first FSOG, (c) an analysis of billing records associated with the first FSOG, (d) an available computing capacity at a machine learning service implemented at a provider network.

9. The method as recited in claim 7 , further comprising:
generating, by the RM, an addition set of PRRs for the first FSOG using a set of resources selected in accordance with a different constraint, wherein the different constraint is determined by the RM subsequent to determining the first constraint.

10. The method as recited in claim 6 , wherein the first machine learning model implements a first modeling methodology selected from a methodology set comprising: (a) regression, (b) classification or (c) time series modeling, and wherein the second machine learning model implements a different modeling methodology of the methodology set.

11. The method as recited in claim 6 , further comprising performing, by the RM:
classifying, using a clustering model, users of the first FSOG into a plurality of categories based at least in part on respective file access patterns of the users, wherein the first set of PRRs is based at least in part on a result of the clustering model.

12. The method as recited in claim 6 , further comprising performing, by the RM:
determining, based at least in part on the first collection of usage metrics, that an application associated with the first FSOG is (a) I/O (input/output)-bound and (b) running at an execution platform with a first computing capacity; and
generating a recommendation to transfer the application to a different execution platform with a different computing capacity.

13. The method as recited in claim 6 , wherein the plurality of SDGs comprises: (a) locally-attached solid state drives (SSD) of instance hosts of the provider network, (b) locally-attached rotating magnetic disk drives of instance hosts of the provider network, (c) network-accessible solid state drives of a service implementing a block-level programmatic interface, (d) network-accessible magnetic disk drives of a service implementing a block-level programmatic interface, (e) devices of an object storage service implementing a web-services interface, (f) devices of a third-party storage service implemented at least in part outside the provider network, or (g) storage de

Assignees

• Amazon Tech Inc

Inventors

• Krishnan Karthikeyan
• Sait Abdul Sathar

Classifications

• G06F17/30079 Primary

  Physics · mapped topic

• G06N99/005

  Physics · mapped topic

• G06F17/30218

  Physics · mapped topic

• G06F17/3097

  Physics · mapped topic

• G06F16/122

  using management policies (point-in-time backing up or restoration of persistent data G06F11/1446; file migration policies for HSM systems G06F16/185) · CPC title