Resource provisioning in database systems

Patent metadata

Field	Value
Publication number	US-11360989-B2
Application number	US-202117462699-A
Country	US
Kind code	B2
Filing date	Aug 31, 2021
Priority date	Apr 10, 2019
Publication date	Jun 14, 2022
Grant date	Jun 14, 2022

Abstract

Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

First claim

What is claimed is:

1. A method for managing and executing database tasks, the method comprising:
determining a set of actions to be executed on a database table in response to a trigger event, the trigger event comprising one or more of an insert, update, or delete operation;
determining a query plan for executing the set of actions, the query plan comprising a plurality of discrete statements to be executed;
determining whether the plurality of statements are to be performed on a row of the database table;
in response to determining whether the plurality of statements are to be performed on the row of the database table, assigning each of the plurality of statements to be performed on the row of the database table to one or more nodes of a plurality of nodes of an execution platform;
determining whether execution of the set of actions is complete; and
storing a record of each completed action of the set of actions.

2. The method of claim 1 , further comprises determining whether the one or more nodes of the plurality of nodes of the execution platform has finished executing the query plan.

3. The method of claim 1 , further comprising retrieving the set of actions from a task queue, wherein the task queue is stored across one or more of a plurality of shared storage devices that are separate from the execution platform.

4. The method of claim 3 , further comprising scaling a number of available execution nodes in the execution platform up or down based on a number of sets of actions waiting in the task queue.

5. The method of claim 1 , further comprising deserializing the task to resolve a context for the task, wherein the context comprises one or more of an account identification, a role, or a user for executing the task.

6. The method of claim 1 , wherein determining the query plan for executing the task comprises generating a Structured Query Language (SQL) job, wherein the plurality of discrete statements must be processed to complete the SQL job.

7. The method of claim 6 , further comprising dividing the processing of the plurality of discrete statements based on metadata, wherein the metadata pertains to database data that is stored across a plurality of shared storage devices, and wherein the metadata is stored separately from the database data.

8. The method of claim 1 , further comprising:
predicting a time required to execute a given statement of the plurality of statements;
determining a current processing delay associated with the one or more nodes based on the predicted time; and
in response to determining that the current processing delay exceeds a threshold value, creating a new node for processing the given statement.

9. The method of claim 1 , wherein assigning each of the plurality of statements to one or more nodes further comprises:
assigning a first statement of the plurality of statements to a first node, the assigning the first statement to the first node comprises determining that the first node includes cached data associated with the first statement; and
assigning a second statement of the plurality of statements to a second node, the second node being distinct from the first node.

10. A system comprising:
a plurality of shared storage devices collectively storing database data;
a compute service manager comprising a software program stored in memory and executed by one or more processors, the compute service manager configured to:
determining a set of actions to be executed on a database table in response to a trigger event, the trigger event comprising one or more of an insert, update, or delete operation;
determining a query plan for executing the set of actions, the query plan comprising a plurality of discrete statements to be executed;
determining whether the plurality of statements are to be performed on a row of the database table;
in response to determining whether the plurality of statements are to be performed on the row of the database table, assigning each of the plurality of statements to be performed on the row of the database table to one or more nodes of a plurality of nodes of an execution platform;
determining whether execution of the set of actions is complete; and
storing a record of each completed action of the set of actions.

11. The system of claim 10 , wherein the compute service manager is further configured for:
determining whether the one or more nodes of the plurality of nodes of the execution platform has finished executing the query plan.

12. The system of claim 11 , wherein the compute service manager is further configured to scale a number of available execution nodes in the execution platform up or down based on a number of sets of tasks within a task queue.

13. The system of claim 10 , wherein the compute service manager is further configured to:
deserialize the statements to resolve a context for the set of actions, wherein the context comprises one or more of an account identification, a role, or a user for executing the set of actions; and
divide processing of the plurality of discrete statements based on metadata, wherein the metadata pertains to the database data and is stored separately from the database data.

14. The method of claim 1 , further comprising prioritizing user queries over internal jobs of the execution platform.

15. The method of claim 1 , further comprising:
determining a future workload of the plurality of nodes; and
scheduling each of the plurality of statements based on the future workload of the plurality of nodes.

16. A non-transitory computer-readable medium comprising instructions, which when executed by at least one processor, cause the at least one processor to perform operations comprising:
determining a set of actions to be executed on a database table in response to a trigger event, the trigger event comprising one or more of an insert, update, or delete operation;
determining a query plan for executing the set of actions, the query plan comprising a plurality of discrete statements to be executed;
determining whether the plurality of statements are to be performed on a row of the database table;
in response to determining whether the plurality of statements are to be performed on the row of the database table, assigning each of the plurality of statements to be performed on the row of the database table to one or more nodes of a plurality of nodes of an execution platform;
determining whether execution of the set of actions is complete; and
storing a record of each completed action of the set of actions.

17. The non-transitory computer-readable medium of claim 16 , the operations further comprises:
determining whether the one or more nodes of the plurality of nodes of the execution platform has finished executing the query plan.

18. The non-transitory computer-readable medium of claim 16 , the operations further comprising:
retrieving the set of actions from a queue, wherein the queue comprises means for storing the set of actions across one or more of a plurality of shared storage devices that are separate from the execution platform.

19. The non-transitory computer-readable medium of claim 18 , the operations further comprising scaling a number of available execution nodes in the execution platform up or down based on a number of sets of actions waiting in the queue.

20. The non-transitory computer-readable medium of claim 16 , the operations further comprising deserializing the statements to resolve a context for the set of actions, wherein the context comprises one or more

Assignees

• Snowflake Inc

Inventors

• Rosen Jeffrey
• Munir Abdul
• Yan Jiaqi
• Waddington William
• Rajaperumal Prasanna
• Cruanes Thierry

Classifications

• G06F16/2465 Primary

  Query processing support for facilitating data mining operations in structured databases · CPC title

• G06F2209/5017

  Task decomposition · CPC title

• G06F16/24565

  Triggers; Constraints · CPC title

• G06F9/4843 Primary

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/5077

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title