Storage lifecycle pipeline architecture

Patent metadata

Field	Value
Publication number	US-10642654-B2
Application number	US-201715671099-A
Country	US
Kind code	B2
Filing date	Aug 7, 2017
Priority date	Dec 14, 2012
Publication date	May 5, 2020
Grant date	May 5, 2020

Abstract

Methods and apparatus for a pipelined architecture for storage lifecycles are disclosed. A method includes identifying, during a transition candidate discovery iteration, storage objects of a multi-tenant service for which respective lifecycle transitions are to be scheduled, including a particular storage object identified as being a candidate for a particular lifecycle transition. The method includes generating transition job objects, each job comprising respective indications of one or more of the storage objects identified during the discovery iteration. The method includes assigning, during a transition execution iteration, resources to initiate storage operations for the particular lifecycle transition, and initiating, using the assigned resources, storage operations for the particular lifecycle transition.

First claim

What is claimed is:

1. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implement a storage lifecycle task dispatcher configured to:
perform a plurality of lifecycle transition execution iterations for storage objects of a multi-tenant storage service, wherein each lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations is initiated based on one or more scheduling criteria;
identify, during a particular lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations, a set of transition job objects to be validated for execution;
validate a particular transition job object of the set; and
in response to validating the particular transition job object of the set:
assign one or more resources to initiate operations to implement lifecycle transitions indicated in the particular transition job object; and
monitor a completion state of an operation to which a particular resource of the one or more resources is assigned.

2. The non-transitory computer-accessible storage medium as recited in claim 1 , wherein a scheduling criterion of the one or more scheduling criteria is based at least in part on one or more of: (a) a predetermined schedule, (b) a determination that a number of transition job objects stored in the storage service exceeds a threshold value, (c) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a transition candidate discovery iteration performed by a transition job generator.

3. The non-transitory computer-accessible storage medium as recited in claim 1 ,
wherein the one or more resources comprise a worker thread, and
wherein the worker thread is configured to:
subsequent to completion of a successful operation, send a task-done notification to the dispatcher, and
for an unsuccessful operation, send a task-failed notification to the dispatcher.

4. The non-transitory computer-accessible storage medium as recited in claim 1 , wherein a particular operation initiated to implement a lifecycle transition indicated in the particular transition job object comprises one or more of: (a) an operation to repartition a storage object according to parameters of an erasure encoding policy or (b) an operation to copy at least a portion of the data of a storage object from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default.

5. The non-transitory computer-accessible storage medium as recited in claim 1 ,
wherein the storage objects of the plurality of lifecycle transition execution iterations are determined by a transition candidate discovery iteration; and,
wherein said perform the plurality of transition execution iterations is performed asynchronously with respect to the transition candidate discovery iteration.

6. The non-transitory computer-accessible storage medium as recited in claim 1 , wherein the program instructions, when executed on the one or more processors further perform delete, after a determination that the storage operations for the particular transition job object have been performed, the particular transition job object from the storage service.

7. A system, comprising:
one or more hardware computing devices comprising one or more respective hardware processors and memory and configured to implement a storage lifecycle task dispatcher configured to:
perform a plurality of lifecycle transition execution iterations for storage objects of a multi-tenant storage service, wherein each lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations is initiated based on one or more scheduling criteria;
identify, during a particular lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations, a set of transition job objects to be validated for execution;
validate a particular transition job object of the set; and
in response to the validation of the particular transition job object of the set:
assign one or more resources to initiate operations to implement lifecycle transitions indicated in the particular transition job object; and
monitor a completion state of an operation to which a particular resource of the one or more resources is assigned.

8. The system as recited in claim 7 , wherein a scheduling criterion of the one or more scheduling criteria is based at least in part on one or more of: (a) a predetermined schedule, (b) a determination that a number of transition job objects stored in the storage service exceeds a threshold value, (c) one or more resource utilization metrics associated with the storage service, or (e) an indication of a status of a transition candidate discovery iteration performed by a transition job generator.

9. The system as recited in claim 7 , wherein the one or more resources comprise a worker thread.

10. The system as recited in claim 7 , wherein a particular operation initiated to implement a lifecycle transition indicated in the particular transition job object comprises one or more of: (a) an operation to repartition a storage object according to parameters of an erasure encoding policy or (b) an operation to copy at least a portion of the data of a storage object from a first storage device configured to remain powered on for continuous access, to a second storage device configured to remain in a low power state by default.

11. The system as recited in claim 7 ,
wherein the storage objects of the plurality of lifecycle transition execution iterations are determined by a transition candidate discovery iteration; and,
wherein said perform the plurality of transition execution iterations is performed asynchronously with respect to the transition candidate discovery iteration.

12. The system as recited in claim 7 , wherein the one or more respective hardware processors and memory are configured to: subsequent to a determination that the storage operations for the particular transition job object have been performed, delete the particular transition job object from the storage service.

13. The system as recited in claim 7 , wherein the one or more resources comprise a worker thread configured to:
subsequent to completion of a successful operation, send a task-done notification to the dispatcher, and
for an unsuccessful operation, send a task-failed notification to the dispatcher.

14. A method, comprising:
performing, by the storage lifecycle task dispatcher implemented via one or more hardware computing devices comprising one or more respective hardware processors and memory, a plurality of lifecycle transition execution iterations for storage objects of a multi-tenant storage service, wherein each lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations is initiated based on one or more scheduling criteria;
identifying, by the storage lifecycle task dispatcher, during a particular lifecycle transition execution iteration of the plurality of lifecycle transition execution iterations, a set of transition job objects to be validated for execution;
validating a particular transition job object of the set; and
in response to said validating the particular transition job object of the set:
assigning one or more resources to initiate operations to implement lifecycle transitions indicated in the particular transition job object; and
monitoring a completion state of an operation to which a particular resource of the o

Assignees

• Amazon Tech Inc

Inventors

• Barber Jeffrey Michael
• Denny-Brown Ii Derek Ernest
• Perry Carl Yates
• Elving Christopher Henning
• Gattu Praveen Kumar

Classifications

• G06F9/5016 Primary

  the resource being the memory · CPC title

• Y02D10/00

  Energy efficient computing, e.g. low power processors, power management or thermal management · CPC title