Managed service for acquisition, storage and consumption of large-scale data streams

Patent metadata

Field	Value
Publication number	US-9794135-B2
Application number	US-201314077173-A
Country	US
Kind code	B2
Filing date	Nov 11, 2013
Priority date	Nov 11, 2013
Publication date	Oct 17, 2017
Grant date	Oct 17, 2017

Abstract

A control node of a multi-tenant stream management service receives a request to initialize a data stream to be comprised of a plurality of data records. The control node determines, based on a partitioning policy, parameters to be used to configure subsystems for ingestion, storage and retrieval of the records. The control node identifies resources to be used for a node of retrieval subsystem The retrieval node is configured to implement programmatic record retrieval interfaces, including respective interfaces to implement non-sequential and sequential access patterns. The control node configures the retrieval node using the selected resources.

First claim

What is claimed is:

1. A system, comprising:
one or more computing devices comprising one or more processors and memory and configured to:
determine, by one or more control components of a multi-tenant stream management service, wherein the one or more control components are assigned to a particular data stream comprising a sequence of a plurality of data records generated by one or more data producers, a respective set of nodes of (a) a record ingestion subsystem, (b) a record storage subsystem, and (c) a record retrieval subsystem, wherein each subsystem of the record ingestion subsystem, the record storage subsystem, and the record retrieval subsystem comprises one or more nodes dynamically configurable by the one or more control components based on one or more policies including a partitioning policy for partitioning the particular data stream that comprises the sequence of the plurality of data records into a plurality of partitions;
receive data records submitted via one or more programmatic record submission interfaces implemented at the record ingestion subsystem, wherein the one or more programmatic record submission interfaces include a first submission interface supporting in-line submission of data records and a second submission interface enabling submission of data records by reference to network addresses at which data is stored;
provide contents of data records in response to data record retrieval requests received via one or more programmatic record retrieval interfaces implemented at the record retrieval subsystem, wherein the one or more programmatic record retrieval interfaces include a first retrieval interface enabling a non-sequential access pattern and a second retrieval interface enabling a sequential access pattern, and wherein a billing rate associated with a use of the first retrieval interface differs from a billing rate associated with a use of the second retrieval interface;
generate a client billing amount associated with the particular data stream based at least in part on respective usage count metrics of the plurality of record retrieval interfaces and the plurality of record submission interfaces.

2. The system as recited in claim 1 , wherein the one or more computing devices are further configured to:
assign, in accordance with the partitioning policy, a particular data record of the particular data stream to a partition of the particular data stream, based at least in part on a key associated with the particular data record, wherein the key is indicated by a write request corresponding to the particular data record.

3. The system as recited in claim 1 , wherein the one or more computing devices are further configured to:
select, based at least in part on a partition to which a particular data record is assigned, one or more of (a) a particular node of the record ingestion subsystem responsible for accepting the particular data record, (b) a particular node of the record storage subsystem responsible for storing at least one copy of the particular data record, and (c) a particular node of the retrieval subsystem responsible for obtaining the particular data record from the record storage subsystem in response to a read request.

4. The system as recited in claim 1 , wherein at least one subsystem of the record ingestion subsystem, the record storage subsystem, and the record retrieval subsystem comprises a plurality of nodes configured by the one or more control components as members of a redundancy group, wherein the redundancy group comprises: (a) one or more primary nodes assigned to perform operations on a set of data records and (b) one or more non-primary nodes configured to assume a primary role in response to a detection of one or more triggering events.

5. The system as recited in claim 1 , wherein the one or more computing devices are further configured to:
generate, corresponding to a particular data record, a particular sequence number indicative of an order in which the particular data record was received at the record ingestion subsystem relative to other data records of the partition to which the particular data record belongs;
store, by the record storage subsystem, a plurality of data records including the particular data record in an order based at least in part on respective sequence numbers generated for the data records; and
in response to a read request invoking the first retrieval interface with the particular sequence number as a parameter, retrieve the particular data record from the record storage subsystem.

6. A method, comprising:
performing, by one or more computing devices:
determining, for a particular data stream comprising a sequence of a plurality of data records, a set of nodes configurable by one or more control components to perform stream management operations based on one or more policies including a stream partitioning policy for partitioning the particular data stream that comprises the sequence of the plurality of data records into a plurality of partitions;
providing data records in response to record retrieval requests received via one or more programmatic record retrieval interfaces, wherein the one or more programmatic record retrieval interfaces include a first retrieval interface enabling a non-sequential access pattern and a second retrieval interface enabling a sequential access pattern, and wherein a billing rate associated with a use of the first retrieval interface differs from a billing rate associated with a use of the second retrieval interface; and
generating a client billing amount associated with the particular data stream based at least in part on respective usage count metrics of the plurality of record retrieval interfaces.

7. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
assigning, in accordance with the partitioning policy, a particular data record of the particular data stream to a first partition of the plurality of partitions of the particular data stream, based at least in part on a key associated with the particular data record, wherein the key is indicated by a write request corresponding to the particular data record.

8. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
selecting, based at least in part on a partition to which a particular data record is assigned, one or more of (a) a particular node of a record ingestion subsystem responsible for accepting the particular data record, (b) a particular node of a record storage subsystem responsible for storing at least one copy of the particular data record, and (c) a particular node of a retrieval subsystem responsible for obtaining the particular data record from the record storage subsystem in response to a read request.

9. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
receiving data records submitted via one or more programmatic record submission interfaces, wherein the one or more programmatic record submission interfaces include a first submission interface supporting in-line submission of data records and a second submission interface enabling submission of data records by reference to one of: (a) an object address at a storage service implemented by a provider network (b) a universal record locator (c) a database record.

10. The method as recited in claim 6 , further comprising performing, by the one or more computing devices:
removing, from a particular node of a data storage subsystem, a particular data record based at least in part on one of: (a) a data de-duplication window configured for the particular data stream, (b) a data archival policy associated with the particular

Assignees

• Amazon Tech Inc

Inventors

• Theimer Marvin Michael
• Ghare Gaurav D
• Dunagan John David
• Burgess Greg
• Xiong Ying

Classifications

• H04L41/5051 Primary

  Service on demand, e.g. definition and deployment of services in real time · CPC title

• G06F17/30516

  Physics · mapped topic

• H04L41/18

  Delegation of network management function, e.g. customer network management [CNM] · CPC title

• H04L41/24 Primary

  using dedicated network management hardware · CPC title

• H04L41/0668

  by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure · CPC title