Data stream ingestion and persistence techniques

Patent metadata

Field	Value
Publication number	US-2018189367-A1
Application number	US-201715858480-A
Country	US
Kind code	A1
Filing date	Dec 29, 2017
Priority date	Nov 11, 2013
Publication date	Jul 5, 2018
Grant date	—

Abstract

A programmatic interface is implemented, enabling a client of a stream management service to select a data ingestion policy for a data stream. A client request selecting an at-least-once ingestion policy is received. In accordance with the at-least-once policy, a client may transmit an indication of a data record one or more times to the service until a positive acknowledgement is received. In response to receiving a plurality of transmissions indicating a particular data record, respective positive acknowledgements are sent to the client. Based on a persistence policy selected for the stream, copies of the data record are stored at one or more storage locations in response to one particular transmission of the plurality of transmissions.

First claim

1 .- 26 . (canceled)

27 . A system, comprising:
one or more computing devices of a stream management service comprising one or more processors and memory and configured to:
receive, from a client of the stream management service, a selection of a data ingestion policy for a particular data stream from among a plurality of data ingestion policies, wherein the data ingestion policies comprise different respective policies for obtaining records from one or more producers;
receive, from the client of the stream management service, a selection of a data persistence policy for the particular data stream from among a plurality of data persistence policies, wherein the data persistence policies comprise different respective policies for storing the records obtained from the one or more producers;
obtain respective data records of the particular data stream from the one or more producers in accordance with the selected data ingestion policy; and
store the obtained data records of the particular data stream in accordance with the selected data persistence policy.

28 . The system as recited in claim 27 , wherein to receive the selection of the data ingestion policy and the data persistence policy for the particular data stream, the one or more computing devices are configured to:
implement one or more programmatic interfaces enabling the client to specify the data ingestion policy and the data persistence policy for the particular data stream.

29 . The system as recited in claim 28 , wherein the one or more computing devices are configured to:
implement the one or more programmatic interfaces as a graphical user interface, a web page, a web site, a command line interface, or an application programming interface.

30 . The system as recited in claim 27 , wherein the one or more computing devices are configured to:
receive, from the client of the stream management service, a selection of a different data ingestion policy for another data stream from among the plurality of data ingestion policies; and obtain respective data records of the other data stream in accordance with the different data ingestion policy.

31 . The system as recited in claim 30 , wherein the one or more computing devices are configured to:
receive, from the client of the stream management service, a selection of a different data persistence policy for the other data stream from among the plurality of data persistence policies; and store the obtained data records of the other data stream in accordance with the different data persistence policy.

32 . The system as recited in claim 27 , wherein the stored data records of the particular data stream respectively comprise different sequence numbers, and wherein the one or more computing devices are configured to generate the different sequence numbers according to an order in which the data records are received at the one or more computing devices.

33 . The system as recited in claim 27 , wherein to receive, from the client of the stream management service, the selection of the data ingestion policy for the particular data stream from among the plurality of data ingestion policies, the one or more computing devices are configured to:
receive a selection of an at-least-once ingestion policy in accordance with which a record submitter transmits an indication of a data record one or more times to the stream management service until a positive acknowledgement is received, or receive a selection of a best-effort ingestion policy, in accordance with which the stream management service is to accept and store at least some data records of a specified stream without providing corresponding positive acknowledgements to a submitter of the at least some data records.

34 . A method, comprising:
performing, by one or more computing devices of a stream management service:
receiving, from a client of the stream management service, a selection of a data ingestion policy for a particular data stream from among a plurality of data ingestion policies, wherein the data ingestion policies comprise different respective policies for obtaining records from one or more producers;
receiving, from the client of the stream management service, a selection of a data persistence policy for the particular data stream from among a plurality of data persistence policies, wherein the data persistence policies comprise different respective policies for storing the records obtained from the one or more producers;
obtaining respective data records of the particular data stream from the one or more producers in accordance with the selected data ingestion policy; and
storing the obtained data records of the particular data stream in accordance with the selected data persistence policy.

35 . The method as recited in claim 34 , further comprising:
implementing one or more programmatic interfaces enabling the client to specify the data ingestion policy and the data persistence policy for the particular data stream.

36 . The method as recited in claim 35 , further comprising:
implementing the one or more programmatic interfaces as a graphical user interface, a web page, a web site, a command line interface, or an application programming interface.

37 . The method as recited in claim 34 , further comprising:
receiving, from the client of the stream management service, a selection of a different data ingestion policy for another data stream from among the plurality of data ingestion policies; and obtaining respective data records of the other data stream in accordance with the different data ingestion policy.

38 . The method as recited in claim 37 , further comprising:
receiving, from the client of the stream management service, a selection of a different data persistence policy for the other data stream from among the plurality of data persistence policies; and storing the obtained data records of the other data stream in accordance with the different data persistence policy.

39 . The method as recited in claim 37 , wherein respective records of the particular data stream comprise a stream identifier for the particular data stream and respective records of the other data stream comprise a different stream identifier for the other data stream.

40 . The method as recited in claim 34 , wherein the stored data records of the particular data stream respectively comprise different sequence numbers, and further comprising:
generating the different sequence numbers according to an order in which the data records are received at the one or more computing devices.

41 . A non-transitory computer-accessible storage medium storing program instructions that, when executed on one or more processors of a stream management service cause the one or more processors to perform:
receiving, from a client of the stream management service, a selection of a data ingestion policy for a particular data stream from among a plurality of data ingestion policies, wherein the data ingestion policies comprise different respective policies for obtaining records from one or more producers; receiving, from the client of the stream management service, a selection of a data persistence policy for the particular data stream from among a plurality of data persistence policies, wherein the data persistence policies comprise different respective policies for storing the records obtained from the one or more producers; obtaining respective data records of the particular data stream from the one or more producers in accordance with the selected data ingestion policy; and storing the obtained data records of the particular data str

Assignees

• Amazon Tech Inc

Inventors

• Theimer Marvin Michael
• Ghare Gaurav D
• Dunagan John David
• Burgess Greg
• Xiong Ying

Classifications

• G06F17/30073

  Physics · mapped topic

• G06F17/30563 Primary

  Physics · mapped topic

• G06F16/113

  Details of archiving (lifecycle management in storage systems G06F3/0649; point-in-time backing up or restoration of persistent data G06F11/1446) · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title