Scalable architecture for a distributed time-series database

Patent metadata

Field	Value
Publication number	US-2020167360-A1
Application number	US-201816199078-A
Country	US
Kind code	A1
Filing date	Nov 23, 2018
Priority date	Nov 23, 2018
Publication date	May 28, 2020
Grant date	—

Abstract

Methods, systems, and computer-readable media for a scalable architecture for a distributed time-series database are disclosed. Using a fleet of ingestion routers, time-series data generated by a plurality of client devices is stored into a plurality of durable partitions. The time-series data comprises a plurality of time series, and an amount of the ingestion routers is determined based at least in part on an ingestion rate of the time-series data. Using a fleet of stream processors, the time-series data from the durable partitions is stored into a plurality of storage tiers including a first storage tier and a second storage tier. A retention period for the first storage tier differs from a retention period for the second storage tier. An amount of the stream processors is determined based at least in part on the time-series data in the durable partitions.

First claim

What is claimed is:

1 . A system, comprising:
a control plane; a fleet of ingestion routers, wherein the fleet of ingestion routers is configured to:
receive time-series data generated by a plurality of client devices, wherein the time-series data is associated with a plurality of time series, and wherein an amount of the ingestion routers is determined by the control plane based at least in part on an ingestion rate of the time-series data; and
store the time-series data using a plurality of durable partitions of a streaming service, wherein the time-series data is partitioned based at least in part on the plurality of time series; and
a fleet of stream processors, wherein the fleet of stream processors is configured to:
store the time-series data from the durable partitions into a plurality of storage tiers including a first storage tier and a second storage tier, wherein a retention period for the first storage tier differs from a retention period for the second storage tier, wherein a performance characteristic for the first storage tier differs from a performance characteristic for the second storage tier, and wherein an amount of the stream processors is determined by the control plane based at least in part on the time-series data in the durable partitions.

2 . The system as recited in claim 1 , further comprising:
a fleet of query processors, wherein the fleet of query processors is configured to:
perform queries of the time-series data stored in the plurality of storage tiers, wherein an amount of the query processors is determined by the control plane based at least in part on the queries.

3 . The system as recited in claim 1 , wherein an amount of the durable partitions is determined by the control plane based at least in part on the time-series data.

4 . The system as recited in claim 1 , wherein an amount of storage resources in the first tier is determined by the control plane based at least in part on an amount of the time-series data within the retention period for the first storage tier, and wherein an amount of storage resources in the second tier is determined by the control plane based at least in part on an amount of the time-series data within the retention period for the second storage tier.

5 . A method, comprising:
storing, by a fleet of ingestion routers into a plurality of durable partitions, time-series data generated by a plurality of client devices, wherein the time-series data is associated with a plurality of time series, and wherein an amount of the ingestion routers is determined based at least in part on an ingestion rate of the time-series data; and storing, by a fleet of stream processors, the time-series data from the durable partitions into a plurality of storage tiers including a first storage tier and a second storage tier, wherein a retention period for the first storage tier differs from a retention period for the second storage tier, and wherein an amount of the stream processors is determined based at least in part on the time-series data in the durable partitions.

6 . The method as recited in claim 5 , further comprising:
performing, by a fleet of query processors, queries of the time-series data stored in the plurality of storage tiers, wherein an amount of the query processors is determined based at least in part on the queries.

7 . The method as recited in claim 5 , wherein an amount of the durable partitions is determined based at least in part on the time-series data.

8 . The method as recited in claim 5 , wherein an amount of storage resources in the first tier is determined based at least in part on an amount of the time-series data within the retention period for the first storage tier, and wherein an amount of storage resources in the second tier is determined based at least in part on an amount of the time-series data within the retention period for the second storage tier.

9 . The method as recited in claim 5 , wherein a latency characteristic for the first storage tier differs from a latency characteristic for the second storage tier.

10 . The method as recited in claim 5 , wherein the time-series data is partitioned into the durable partitions based at least in part on a hierarchy of the time series.

11 . The method as recited in claim 5 , wherein the time-series data is stored in the first storage tier using a plurality of tiles, wherein the tiles are partitioned based at least in part on spatial boundaries and temporal boundaries.

12 . The method as recited in claim 5 , further comprising:
organizing, by the fleet of stream processors, the time-series data from the durable partitions into a plurality of tables, wherein the tables are stored in the plurality of storage tiers; and transforming, by the fleet of stream processors, the time-series data from the tables into a plurality of additional tables, wherein the additional tables are stored in the plurality of storage tiers.

13 . The method as recited in claim 12 , wherein the time-series data within a window of time is transformed using aggregation into the additional tables, wherein the window of time is determined according to input to a control plane.

14 . One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more processors, perform:
storing, by a fleet of ingestion routers into a plurality of durable partitions, time-series data generated by a plurality of client devices, wherein the time-series data is associated with a plurality of time series, and wherein an amount of the ingestion routers is determined by a control plane based at least in part on an ingestion rate of the time-series data; storing, by a fleet of stream processors, the time-series data from the durable partitions into a plurality of storage tiers including a first storage tier and a second storage tier, wherein a retention period for the first storage tier differs from a retention period for the second storage tier, and wherein an amount of the stream processors is determined by the control plane based at least in part on the time-series data in the durable partitions; and performing, by a fleet of query processors, queries of the time-series data stored in the plurality of storage tiers, wherein an amount of the query processors is determined by the control plane based at least in part on the queries.

15 . The one or more non-transitory computer-readable storage media as recited in claim 14 , wherein the queries are performed using one or more indices of the time-series data, wherein the one or more indices represent timestamps of the time-series data, the plurality of time series, and the plurality of storage tiers.

16 . The one or more non-transitory computer-readable storage media as recited in claim 14 , wherein an amount of the durable partitions is determined by the control plane based at least in part on the time-series data.

17 . The one or more non-transitory computer-readable storage media as recited in claim 14 , wherein an amount of storage resources in the first tier is determined by the control plane based at least in part on an amount of the time-series data within the retention period for the first storage tier, and wherein an amount of storage resources in the second tier is determined by the control plane based at least in part on an amount of the time-series data within the retention period for the second storage tier.

18 . The one or more non-transitory computer-readable storage media as recited in claim 14 , wherein the time-series

Assignees

• Amazon Tech Inc

Inventors

• Rath Timothy A
• Gupta Gaurav
• Ozen Mustafa Ozan
• Zaki Omer Ahmed

Classifications

• G06F16/252

  between a Database Management System and a front-end application · CPC title

• G06F16/278

  Data partitioning, e.g. horizontal or vertical partitioning · CPC title

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0607

  by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device · CPC title