Lifecycle transitions in log-coordinated data stores

Patent metadata

Field	Value
Publication number	US-10373247-B2
Application number	US-201414491444-A
Country	US
Kind code	B2
Filing date	Sep 19, 2014
Priority date	Sep 19, 2014
Publication date	Aug 6, 2019
Grant date	Aug 6, 2019

Abstract

Based at least in part on an optimization goal determined for a storage group comprising a plurality of data stores for which writes requests are coordinated by a log-based transaction manager, one or more lifecycle transitions are identified for data objects stored in a first data store. In accordance with a first lifecycle transition, a data object is to be transferred from the first data store to a second data store based at least in part on a duration for which the data object has been resident at the first data store. A transition agent is assigned by a configuration manager to implement the first lifecycle transition.

First claim

What is claimed is:

1. A system, comprising:
one or more processors to perform instructions of a configuration manager of a heterogeneous storage service to:
determine one or more lifecycle transitions to be performed for at least a portion of a data set to be stored at a first data store of a plurality of data stores of a log-coordinated storage group established on behalf of a client, wherein, in accordance with a first lifecycle transition of the one or more lifecycle transitions, a data object that has been accepted for inclusion in the first data store is to be transferred to a second data store of the plurality of data stores based at least in part on one or more time-based transition criteria, wherein at least one of the time-based transition criteria comprises a delay since the object was accepted for inclusion in the first data store, and wherein the second data store is designated as a destination of the first lifecycle transition based at least in part on a difference in service capability of the first data store and the second data store; and
assign a particular transition agent of one or more transition agents to implement the first lifecycle transition; and

one or more processors to perform instructions of the particular transition agent, responsive to assignment of the particular transition agent to implement the first lifecycle transition, to:
responsive to detection that a particular data object of the first data store meets a particular time-based transfer criterion of the one or more time-based transition criteria, transfer the particular data object to the second data store in accordance with the first lifecycle transition to provide a change of service capability for the particular data object; and
store a transition record indicating that the particular data object has been transferred.

2. The system as recited in claim 1 , wherein the configuration manager is configured to:
receive a lifecycle flow request from the client via a programmatic interface, wherein the lifecycle flow request indicates the first data store, the second data store, and the one or more time-based transfer criterion.

3. The system as recited in claim 2 , wherein the configuration manager is configured to:
receive an indication of a optimization goal of the client; and
identify the first lifecycle transition as a candidate lifecycle transition based at least in part on the optimization goal.

4. The system as recited in claim 1 , wherein the difference in service capability comprises a difference in one or more of: (a) a data model, (b) a data isolation level, (c) availability, (d) data durability, or (e) performance.

5. The system as recited in claim 1 , further comprising a transaction manager of the log-coordinated storage group, wherein the transaction manager is configured to:
receive a transaction request from the client, indicating (a) a particular write operation directed to the second data store and (b) a read descriptor indicating one or more reads on which the particular write depends;
determine, based at least in part on the read descriptor and at least in part on one or more write records stored in a persistent log, that the particular write is to be accepted for commit; and
initiate a propagation of the particular write to the second data store, without propagating the write to the first data store.

6. A method, comprising:
determining, at a configuration manager of a heterogeneous storage service, one or more lifecycle transitions to be implemented for at least a portion of a first data store of a plurality of data stores for which write requests are coordinated by a log-based transaction manager on behalf of a client, wherein, in accordance with a first lifecycle transition of the one or more lifecycle transitions, a data object is to be transferred to a second data store of the plurality of data stores based at least in part on a transfer criterion, wherein the transition criterion comprises a duration for which the data object has resided in the first data store, and wherein the second data store is selected as a destination of the first lifecycle transition based at least in part on a difference in a service capability between the first data store and the second data store;
assigning, by the configuration manager, a particular transition agent of one or more transition agents of the heterogeneous storage service to perform the first lifecycle transition; and
in response to detecting, by the particular transition agent and in accordance with the first lifecycle transition, that a particular data object meets the transfer criterion, transferring, by the particular transition agent, the particular data object to the second data store and storing, by the particular transition agent, a transition record indicating that the particular data object has been transferred in accordance with the first lifecycle transition to provide a transition of service capability on behalf of the client.

7. The method as recited in claim 6 , wherein the difference in service capability comprises a difference in one or more of: (a) a data model, (b) a data isolation level, (c) availability, (d) data durability, or (e) performance.

8. The method as recited in claim 6 , wherein in accordance with the first lifecycle transition, the data object is to be transferred to the second data store based at least in part on a difference in a first pricing policy associated with the first data store, and a second pricing policy associated with the second data store.

9. The method as recited in claim 6 , wherein the first data store includes a plurality of storage devices utilizing a first medium type of a set of medium types comprising (a) main memory-resident storage, (b) solid-state-drive storage, (c) rotating-disk-based storage and (d) tape-based storage, and wherein the second data store includes at least one storage device utilizing the first medium type.

10. The method as recited in claim 6 , wherein the particular data object is stored at a first storage device of the first data store at a first data center of a provider network, and wherein the particular data object is transferred by the particular transition agent to a different storage device of the second data store at a different data center.

11. The method as recited in claim 6 , further comprising:
receiving, by the configuration manager, a lifecycle flow request from the client via a programmatic interface, wherein the lifecycle flow request indicates the first data store, the second data store, and the transfer criterion.

12. The method as recited in claim 11 , further comprising:
receiving, by the configuration manager, an indication of a optimization goal of the client; and
identifying, by the configuration manager, the first lifecycle transition as a candidate lifecycle transition based at least in part on the optimization goal.

13. The method as recited in claim 6 , further comprising:
generating, by the particular transition agent to transfer the particular data object to the second data store, a representation of the particular data object based at least in part on an examination of a commit record associated with the particular data object, wherein the commit record is stored in a persistent write log of the log-based transaction manager, wherein the persistent write log is implemented as a replication graph.

14. The method as recited in claim 6 , wherein the first data store comprises one of: a non-relational database system, a relational database system, a storage service that implements a web services interface allowing access to unstructured data objects, an in-memory d

Assignees

• Amazon Tech Inc

Inventors

• Vermeulen Allan Henry
• Zaki Omer Ahmed

Classifications

• G06Q40/00 Primary

  Finance; Insurance; Tax strategies; Processing of corporate or income taxes · CPC title