Materialization strategies in journal-based databases

Patent metadata

Field	Value
Publication number	US-10133767-B1
Application number	US-201514868271-A
Country	US
Kind code	B1
Filing date	Sep 28, 2015
Priority date	Sep 28, 2015
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

A journal manager of a multi-data-store storage appends a committed transaction entry to a journal, indicating a state change which has been approved for commit using an optimistic concurrency control algorithm. A first representation of the state change is generated and stored at one materialization node. A different representation of the same state change, comprising at least one attribute for which a value is not included in the first representation, is generated and stored at a different materialization node.

First claim

What is claimed is:

1. A system, comprising:
a journal manager implemented at one or more computing devices associated with a multi-data-store database;
a first materialization node of the multi-data-store database; and
a second materialization node of the multi-data-store database;
wherein the journal manager is configured to:
append, after performing commit analysis on a particular transaction request in accordance with an optimistic concurrency control protocol, a committed transaction entry to a journal, wherein the particular transaction request is expressed in a data-store-independent transaction language specified in a journal schema of the multi-data-store database, and wherein the committed transaction entry indicates that a state change proposed in the particular transaction request has been accepted for commit;

wherein the first materialization node is configured to:
generate, according to a first materialization strategy corresponding to a first category of queries supported at the first materialization node, a first representation of the state change;
store, at a storage device of the first materialization node, the first representation; and

wherein the second materialization node is configured to:
generate, according to a second materialization strategy corresponding to a second category of queries supported at the second materialization node, a second representation of the state change, wherein the second representation includes a particular attribute which is not included in the first representation, wherein the particular attribute indicates a logical relationship between data objects materialized at the second materialization node;
store, at a storage device of the second materialization node, the second representation.

2. The system as recited in claim 1 , wherein the first materialization strategy indicates one or more of: (a) a first set of attributes defined in the journal schema, wherein respective values of each attribute of the first set are to be stored at the first materialization node, (b) a second set of derived attributes, wherein respective values of each attribute of the second set are to be stored at the first materialization node, or (c) one or more transformation functions to be applied to generate a representation of a state change to be stored at the first materialization node.

3. The system as recited in claim 1 , wherein the particular attribute comprises a timestamp value associated with the state change.

4. The system as recited in claim 1 , wherein the first representation comprises a parent-pointer indicating a hierarchical relationship between a pair of data objects, and wherein the second representation comprises a row of a closure table indicating the hierarchical relationship.

5. The system as recited in claim 1 , further comprising one or more control plane components of the multi-data-store database, wherein the one or more control plane components are configured to:
register, in response to a request received via a programmatic interface, after the first representation has been stored at the first materialization node, a third materialization node as a consumer of the journal, wherein the third materialization node is responsible for implementing a third materialization strategy.

6. A method, comprising:
appending, by a journal manager implemented at one or more computing devices of a multi-data-store storage system after performing a commit analysis operation on a transaction request in accordance with an optimistic concurrency control protocol, a particular committed transaction entry to a journal, wherein the transaction request is expressed in a data-store-independent transaction language, and wherein the particular committed transaction entry indicates one or more writes directed to one or more data objects;
generating, at a first materialization node comprising one or more computing devices of the multi-data-store storage system, a first representation of the one or more writes of the particular committed transaction entry;
storing, at a storage device of the first materialization node, the first representation;
generating, at a second materialization node comprising one or more computing devices of the multi-data-store storage system, a second representation of the one or more writes of the particular committed transaction entry, wherein the second representation includes a value of a particular attribute which is not included in the first representation; and
storing, at a storage device of the second materialization node, the second representation.

7. The method as recited in claim 6 , wherein said storing the second representation comprises storing a new timestamped version of a first data object without overwriting a previous version of the first data object, and wherein said storing the first representation comprises overwriting a previous version of the first data object.

8. The method as recited in claim 6 , wherein said storing the first representation comprises storing a parent-pointer to indicate a hierarchical relationship between a pair of data objects, and wherein said storing the second representation comprises modifying one or more rows of a closure table to indicate the hierarchical relationship.

9. The method as recited in claim 6 , wherein said generating the first representation comprises applying a transformation function to at least a portion of a first data object, and wherein said generating the second representation does not comprise applying the transformation function.

10. The method as recited in claim 6 , further comprising:
detecting, by a write applier of a third materialization node of the multi-data-store storage system, that another committed transaction entry appended to the journal by the journal manager is directed to a first partition of a plurality of partitions of a data object, wherein said detecting is based at least in part on a value of a particular attribute indicated in the other committed transaction entry; and
generating, at the third materialization node according to a particular materialization strategy associated with the first partition, a particular representation of a write indicated in the other committed transaction entry; and
storing, at a storage device of the third materialization node, the particular representation.

11. The method as recited in claim 6 , further comprising:
registering, by one or more control-plane components of the multi-data-store storage system in response to a request received via a programmatic interface, after the first representation has been stored at the first materialization node, a third materialization node;
generating, at the third materialization node after said registering, a third representation of the one or more writes; and
storing, at a storage device of the third materialization node, the third representation.

12. The method as recited in claim 6 , further comprising:
examining, by a first write applier of the first materialization node prior to said generating the first representation, the particular committed transaction entry, as part of the first write applier's sequential analysis of the journal; and
examining, by a second write applier of the second materialization node, asynchronously with respect to the first write applier, the particular committed transaction entry as part of the second write applier's sequential analysis of the journal.

13. The method as recited in claim 6 , wherein the data-store independent transaction language is defined in a journal schema of the multi-data-store storage system.

14. The method as recited in claim 6 , wherein the j

Assignees

• Amazon Tech Inc

Inventors

• Cole Timothy Daniel
• Danilov Artem
• Ross Andrew Wayne
• Morkel John Michael
• Certain Tate Andrew
• Vermeulen Allan Henry
• Jacques De Kadt Christopher Richard

Classifications

• G06F17/30424

  Physics · mapped topic

• G06F17/30353 Primary

  Physics · mapped topic

• G06F17/30371

  Physics · mapped topic

• G06F17/30383

  Physics · mapped topic

• G06F17/30356

  Physics · mapped topic