Scalable transaction-based data repository service

Patent metadata

Field	Value
Publication number	US-9747288-B1
Application number	US-201213710236-A
Country	US
Kind code	B1
Filing date	Dec 10, 2012
Priority date	Dec 10, 2012
Publication date	Aug 29, 2017
Grant date	Aug 29, 2017

Abstract

A reliable and scalable data repository service can be supporting cooperative transactions. In an example, multiple data producers may use the data repository service to upload data in one or more transactions. Data contained in one transaction may be treated as an atomic unit. In an embodiment, the data repository service manages multiple candidate transactions associated with an election transaction such that at most one candidate transaction may begin and/or commit a given time. During a commit of a candidate transaction, the data uploaded during the candidate transaction may become associated with the election transaction. The data repository service may maintain metadata associated with each of the candidate transactions. In response to a request for data associated with the election transaction, the metadata may be used, in conjunction with a user-specified isolation level, to locate the data uploaded some of the candidate transactions for the election transaction.

First claim

What is claimed is:

1. A computer-implemented method for providing a data repository service, comprising:
managing, by one or more computer systems, start of a plurality of candidate transactions associated with an election transaction by at least updating metadata associated with at least some of the plurality of candidate transactions to ensure consistency, such that no more than one candidate transaction of the plurality of candidate transactions starts at any given time and at least two candidate transactions of the plurality of candidate transactions are executed at least in part concurrently;
storing, by the one or more computer systems, a plurality of candidate datasets respectively associated with the plurality of candidate transactions, each candidate dataset of the plurality of candidate datasets being provided in one or more data batches where each data batch of the one or more data batches includes a subset of the candidate dataset;
for each candidate transaction of the plurality of candidate transactions, maintaining, by the one or more computer systems, the metadata associated with the candidate transaction, the metadata including storage location information and state information for the candidate dataset associated with the candidate transaction;
managing, by the one or more computer systems, commits of the plurality of candidate transactions by at least updating the metadata associated with at least some of the plurality of candidate transactions to ensure consistency, such that no more than one candidate transaction of the plurality of candidate transactions commits at any given time;
receiving, by the one or more computer systems, a request for data resulting from the election transaction, the request comprising an isolation level that is associated with a commit state corresponding to the state information of the metadata associated with each candidate transaction;
selecting, by the one or more computer systems, one or more of the plurality of candidate datasets based at least in part on the isolation level and the metadata associated with one or more of the plurality of candidate transactions corresponding to the one or more of the plurality of candidate datasets; and
providing, by the one or more computer systems, requested data based at least in part on the one or more selected candidate datasets.

2. The computer-implemented method of claim 1 , wherein at least some of the plurality of candidate transactions are each executed by a distinct data producer.

3. The computer-implemented method of claim 2 , wherein the data producer includes a Hadoop node.

4. The computer-implemented method of claim 1 , wherein the isolation level includes at least one of snapshot isolated, repeatable snapshot isolated or read uncommitted.

5. A computer-implemented method for managing transactions, comprising:
managing, by a data repository service, a plurality of transactions by at least maintaining metadata associated with at least some of the plurality of transactions to ensure data consistency, wherein the metadata includes storage location information and state information for a dataset associated with the at least some of the plurality of transactions, wherein individual transactions are executed by a computing entity and are related to a common task, and wherein at least some of the plurality of transactions are executed concurrently at least in part, such that no more than one transaction of the plurality of transactions begins at any given time;
managing, by the data repository service, commits of the plurality of transactions by at least updating the metadata associated with the at least some of the plurality of transactions, such that no more than one transaction of the plurality of transactions commits at any given time;
receiving, by the data repository service, a request for data resulting from the at least some of the plurality of transactions, the request being associated with an isolation level, wherein the isolation level is associated with a commit state corresponding to the state information of the metadata associated with the at least some of the plurality of transactions;
selecting, by the data repository service, the dataset associated with the at least some a subset of the plurality of transactions as data associated with the common task based at least in part on the isolation level and the metadata associated with the at least some of the plurality of transactions; and
providing, by the data repository service, the requested data based at least in part on the selected dataset.

6. The computer-implemented method of claim 5 , wherein the common task involves uploading data to the data repository service.

7. The computer-implemented method of claim 5 , wherein the common task involves removing data from the data repository service.

8. The computer-implemented method of claim 5 , wherein execution of an individual transaction includes uploading one or more batches of data to the data repository service.

9. The computer-implemented method of claim 8 , wherein the metadata is further associated with each batch of one or more batches of data uploaded in each transaction of the plurality of transactions.

10. The computer-implemented method of claim 9 , wherein selecting the dataset associated with a subset of the plurality of transactions comprises determining a location of the one or more batches of data uploaded in each transaction of the subset of the plurality of transactions based at least in part on the metadata associated with the one or more batches.

11. The computer-implemented method of claim 5 , wherein the at least some of the plurality of transactions are executed to improve performance characteristics associated with performance of the common task.

12. A computer system for providing a data repository service, comprising:
one or more processors; and
memory, including instructions executable by the one or more processors to cause the computer system to at least:
manage a plurality of election transactions, wherein an individual election transaction of the plurality of election transactions is implemented by a plurality of candidate transactions, wherein the plurality of candidate transactions are executed collectively by a plurality of computing entities, wherein the plurality of election transactions are managed by modifying metadata associated with the plurality of candidate transactions to ensure consistency, and wherein at least two of the plurality of candidate transactions are executed concurrently at least in part;
for individual election transactions of the plurality of election transactions:
store a plurality of candidate datasets respectively associated with the corresponding plurality of candidate transactions, wherein an individual candidate dataset of the plurality of candidate datasets is provided in one or more data batches where an individual data batch of the one or more data batches includes a subset of the candidate dataset; and
allow no more than one of the plurality of candidate transactions to commit at any given time;
maintain the metadata associated with the plurality of candidate datasets associated with the plurality of candidate transactions, wherein the metadata ensures data isolation between the plurality of candidate transactions, and wherein the metadata comprises storage location information and state information for the plurality of candidate datasets;
receive a request for data resulting from the election transaction, the request comprising an isolation level that is associated with a commit state corresponding to the state information of the metadata;
select one or more of the plurality of candidate data

Assignees

• Amazon Tech Inc

Inventors

• Beckford Nathaniel Jonah
• Choi Seung Hyok
• White John Kenneth

Classifications

• G06F16/2308

  Concurrency control (transaction processing G06F9/466) · CPC title

• G06F16/2379 Primary

  Updates performed during online database operations; commit processing · CPC title

• G06F17/30008 Primary

  Physics · mapped topic

• G06F16/2386 Primary

  Bulk updating operations (data conversion details G06F16/258) · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title