Method and system for designing a database system for high event rate, while maintaining predictable query performance

Patent metadata

Field	Value
Publication number	US-9875272-B1
Application number	US-201514748225-A
Country	US
Kind code	B1
Filing date	Jun 23, 2015
Priority date	Jun 23, 2015
Publication date	Jan 23, 2018
Grant date	Jan 23, 2018

Abstract

A computer-implemented database system, method, and computer program product are provided. the database system comprises a plurality of nodes, each node including a data storage device having memory and a server; and a database system controller coupled to each of the plurality of nodes and having a processor and software program code for directing the database system to perform the following function of defining, for a user, a node group including at least one of the plurality of nodes, upon which the user's data is to be stored in a user database; wherein the user database is defined in terms of time-partitioned tables residing on the nodes of the node group, and wherein each time-partitioned table is further defined in terms of shards of the user's data, the shards corresponding with respective ones of the time partitions. The user data is managed in terms of the shards on respective nodes of the node group, and the time-partitioned tables include a chronologically oldest time-partitioned table and a current time-partitioned table into which a new user data entry will be written. The system further performs the functions of determining whether a predetermined criterion has been met; and, responsive to the detecting that the predetermined criterion has been met, purging all shards of the chronologically oldest time-partitioned table.

First claim

What is claimed is:

1. A computer-implemented database system comprising:
a plurality of nodes, each node including a data storage device having memory and a server, each node of the plurality of nodes being listed in a naming registry service for distributed configuration and synchronization of distributed systems;
a database system controller coupled to each of the plurality of nodes and having a processor and software program code for directing the database system to perform the functions of:
defining, for a user, a node group including at least one of the plurality of nodes, upon which the user's data is to be stored in a user database,
wherein the user database is defined in terms of time-partitioned tables residing on the nodes of the node group,
wherein each time-partitioned table is further defined in terms of shards of the user's data, the shards corresponding with respective ones of the time partitions, and wherein user data is managed in terms of the shards on respective nodes of the node group, and
wherein the time-partitioned tables include a chronologically oldest time-partitioned table and a current time-partitioned table, into which a new user data entry will be written;
determining whether a predetermined criterion has been met;
responsive to the detecting that the predetermined criterion has been met, purging all shards of the chronologically oldest time-partitioned table,
adding an additional node to the plurality of nodes responsive to (i) receipt of additional data to store and further responsive to (ii) detection that additional storage is needed;
registering the additional node with the naming registry service for distributed configuration and synchronization of distributed systems; and
treating the additional node as part of a time partition commencing as of the time the additional node is added to the plurality of nodes and registered with the naming registry service for distributed configuration and synchronization of distributed systems,
wherein the received additional data is stored in the additional node without remapping data that is already in the plurality of nodes of the node group.

2. A computer-implemented database system as recited in claim 1 , wherein the naming registry service for distributed configuration and synchronization of distributed systems is the Zookeeper software project of the Apache Software Foundation.

3. A computer-implemented database system as recited in claim 1 , wherein processing a database query against a database supported by the system includes:
selecting, by start time and end time, time partitions needed to answer the query;
obtaining base information from the selected time partitions;
aggregating the obtained base information; and
returning the aggregated base information as the answer to the query.

4. A computer-implemented database system as recited in claim 1 , further comprising:
maintaining a maximum user data capacity, defined in terms of memory allocated to the user and partitioned into the shards;
wherein the determining whether a predetermined criterion has been met comprises detecting when the current shard is full.

5. A computer-implemented database system as recited in claim 1 , further comprising:
wherein the determining whether a predetermined criterion has been met comprises detecting when the chronologically oldest shard has reached a predetermined age threshold.

6. A computer-implemented database system as recited in claim 1 , further comprising processing a query by aggregating obtained data entries and returning the aggregated data entries as the result.

7. A computer-implemented database system as recited in claim 1 , further comprising classifying the user into one of a plurality of customer classes.

8. A computer-implemented database system as recited in claim 7 , wherein the customer classes are defined in terms of complexity of queries that will be used to access the customer's data.

9. A method for operating a computer-implemented database system comprising:
defining, for a user, a node group including at least one of a plurality of nodes, upon which the user's data is to be stored in a user database,
wherein the user database is defined in terms of time-partitioned tables residing on the nodes of the node group,
wherein each time-partitioned table is further defined in terms of shards of the user's data, the shards corresponding with respective ones of the time partitions, and wherein user data is managed in terms of the shards on respective nodes of the node group, and
wherein the time-partitioned tables include a chronologically oldest time-partitioned table and a current time-partitioned table, into which a new user data entry will be written;
determining whether a predetermined criterion has been met;
responsive to the detecting that the predetermined criterion has been met, purging all shards of the chronologically oldest time-partitioned table;
adding an additional node to the plurality of nodes responsive to (i) receipt of additional data to store and further responsive to (ii) detection that additional storage is needed;
registering the additional node with a naming registry service for distributed configuration and synchronization of distributed systems; and
treating the additional node as part of a time partition commencing as of the time the additional node is added to the plurality of nodes and registered with the naming registry service for distributed configuration and synchronization of distributed systems,
wherein the received additional data is stored in the additional node without remapping data that is already in the plurality of nodes of the node group.

10. A method as recited in claim 9 , wherein the naming registry service for distributed configuration and synchronization of distributed systems is the Zookeeper software project of the Apache Software Foundation.

11. A method as recited in claim 9 , wherein processing a database query against a database supported by the system includes:
selecting, by start time and end time, time partitions needed to answer the query;
obtaining base information from the selected time partitions;
aggregating the obtained base information; and
returning the aggregated base information as the answer to the query.

12. A method as recited in claim 9 , further comprising:
maintaining a maximum user data capacity, defined in terms of memory allocated to the user and partitioned into the shards;
wherein the determining whether a predetermined criterion has been met comprises detecting when the current shard is full.

13. A method as recited in claim 9 , further comprising:
wherein the determining whether a predetermined criterion has been met comprises detecting when the chronologically oldest shard has reached a predetermined age threshold.

14. A method as recited in claim 9 , further comprising processing a query by aggregating obtained data entries and returning the aggregated data entries as the result.

15. A method as recited in claim 9 , further comprising classifying the user into one of a plurality of customer classes.

16. A method as recited in claim 15 , wherein the customer classes are defined in terms of complexity of queries that will be used to access the customer's data.

17. A computer program product for directing a computer system to operate a database system, the computer program product comprising:
a non-transitory computer-readable medium; and
computer program software code, provided on the non-transitory computer-readable medium, for directing a computer-implemented data

Assignees

• Google Inc

Inventors

• Jhingran Anant
• Kumar Rupesh
• Bose Sanjoy

Classifications

• G06F17/30581

  Physics · mapped topic

• G06F17/30342

  Physics · mapped topic

• G06F17/30371 Primary

  Physics · mapped topic

• G06F17/30584

  Physics · mapped topic

• G06F16/278

  Data partitioning, e.g. horizontal or vertical partitioning · CPC title