Automated configuration of log-coordinated storage groups

Patent metadata

Field	Value
Publication number	US-11397709-B2
Application number	US-201816035425-A
Country	US
Kind code	B2
Filing date	Jul 13, 2018
Priority date	Sep 19, 2014
Publication date	Jul 26, 2022
Grant date	Jul 26, 2022

Abstract

A configuration manager of a storage service receives a set of service requirements, comprising one or more of: a performance requirement for one or more types of storage operations, or an access interface type requirement Based on the service requirements, a candidate storage configuration that includes one or more data store instances and a first log-based transaction manager is generated. Subsequent to an approval of the first storage configuration by a client, the establishment of the data store instances and the log-based transaction manager is initiated.

First claim

What is claimed is:

1. A system, comprising:
one or more computing devices to:
receive configuration data from a client to create a log-coordinated storage group (LCSG) in a provider network, the configuration data indicating a plurality of data stores as members of the LCSG;
create the LC SG in the provider network, wherein the LCSG is configured according to the configuration data to:
determine, by a conflicts detector, whether requested transactions are accepted for commit based at least in part on a detection of conflicts between individual ones of the requested transactions and data written by one or more previously-committed transactions indicated in a transaction log;
in response to a determination by the conflicts detector that a requested transaction is accepted for commit, cause the requested transaction to be written to the transaction log; and
propagate committed transactions from the transaction log to at least some of the data stores; and

provide access metadata for the LCSG to the client.

2. The system of claim 1 , further comprising a configuration designer to:
determine a plurality of service requirements of the client for the LCSG, wherein the plurality of service requirements include one or more of: (a) a performance requirement for one or more types of storage operations to be performed at a storage group, (b) an availability requirement, (c) a data durability requirement, or (d) an access interface requirement;
provide, to the client, an indication of one or more candidate storage group configurations that can meet at least a subset of the plurality of requirements; and
receive an indication of approval by the client of one of the candidate storage configurations;
wherein the LCSG is created using the one candidate storage configuration.

3. The system of claim 1 , wherein the configuration data indicates, in the plurality of data stores, at least two data stores of different data store types selected from: a relational database, a non-relational database, an unstructured data store, a storage service, an in-memory cache, or a distributed cache.

4. The system of 1 , wherein:
the configuration data specifies a write applier to propagate data from the transaction log to one of the data stores; and
to create the LCSG, the one or more computing devices is to launch an instance of the write applier in the provider network.

5. The system of 1 , wherein:
the configuration data specifies a write transformer to transform data from one of the data stores and write the transformed data to another one of the data stores; and
to create the LCSG, the one or more computing devices is to launch an instance of the write transformer in the provider network.

6. The system of 5 , wherein:
to create the LCSG, the one or more computing devices is to register the instance of the write transformer as a listener to be notified when a write is applied to the one data store.

7. The system of 5 , wherein:
the configuration data specifies, for the write transformer, one or more time-based trigger conditions to invoke the write transformer.

8. The system of 5 , wherein:
the configuration data specifies computer-readable code that extends functionality of the write transformer.

9. The system of 1 , wherein the one or more computing devices to:
receive further configuration data to add or remove a data store as a member of the LCSG; and
modify the LCSG to add or remove the data store according to the further configuration data.

10. A computer-implemented method, comprising:
receiving configuration data from a client to create a log-coordinated storage group (LCSG) in a provider network, the configuration data indicating a plurality of data stores as members of the LCSG;
creating the LCSG in the provider network and configuring the LCSG to according to the configuration data to perform:
determining, by a conflicts detector, whether requested transactions are accepted for commit based at least in part on a detection of conflicts between individual ones of the requested transactions and data written by one or more previously-committed transactions indicated in a transaction log;
in response to a determination by the conflicts detector that a requested transaction is accepted for commit, cause the requested transaction to be written to the transaction log; and
propagating committed transactions from the transaction log to at least some of the data stores; and

providing access metadata for the LCSG to the client.

11. The method of claim 10 , wherein the configuration data indicates, in the plurality of data stores, at least two data stores of different data store types selected from: a relational database, a non-relational database, an unstructured data store, a storage service, an in-memory cache, or a distributed cache.

12. The method of 10 , wherein:
the configuration data specifies a write applier to propagate data from the transaction log to one of the data stores; and
creating the LCSG comprises launching an instance of the write applier in the provider network.

13. The method of 10 , wherein:
the configuration data specifies a write transformer to transform data from one of the data stores and write the transformed data to another one of the data stores; and
creating the LCSG comprises launching an instance of the write transformer in the provider network.

14. The method of 13 , wherein:
creating the LC SG comprises registering the instance of the write transformer as a listener to be notified when a write is applied to the one data store.

15. The method of 10 , wherein the one or more computing devices to:
receiving further configuration data to add or remove a data store as a member of the LCSG; and
modify the LCSG to add or remove the data store according to the further configuration data.

16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors cause the one or more processors to:
receive configuration data from a client to create a log-coordinated storage group (LCSG) in a provider network, the configuration data indicating a plurality of data stores as members of the LCSG;
cause the LCSG to be created in the provider network, wherein the LCSG is configured according to the configuration data to:
determine, by a conflicts detector, whether requested transactions are accepted for commit based at least in part on a detection of conflicts between individual ones of the requested transactions and data written by one or more previously-committed transactions indicated in a transaction log;
in response to a determination by the conflicts detector that a requested transaction is accepted for commit, cause the requested transaction to be written to the transaction log; and
propagate committed transactions from the transaction log to at least some of the data stores; and

provide access metadata for the LC SG to the client.

17. The non-transitory computer-accessible storage medium of claim 16 , wherein the program instructions when executed on the one or more processors cause the one or more processors to:
determine a plurality of service requirements of the client for the LCSG, wherein the plurality of service requirements include one or more of: (a) a performance requirement for one or more types of storage operations to be performed at a storage group, (b) an availability requirement, (c) a data durability requirement, or (d) an access interface requirement;
provide, to the client, an indication of one or more candidate storage group configurations t

Assignees

• Amazon Tech Inc

Inventors

• Vermeulen Allan Henry
• Zaki Omer Ahmed
• Shevade Upendra Bhalchandra
• Rogers Gregory Rustin

Classifications

• G06F16/21 Primary

  Design, administration or maintenance of databases · CPC title