Resource management systems and methods

Patent metadata

Field	Value
Publication number	US-10366102-B2
Application number	US-201414518884-A
Country	US
Kind code	B2
Filing date	Oct 20, 2014
Priority date	Feb 19, 2014
Publication date	Jul 30, 2019
Grant date	Jul 30, 2019

Abstract

Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

First claim

The invention claimed is:

1. A system comprising:
a plurality of shared storage devices collectively storing database data comprising files; a resource manager comprising a software program stored in memory and executed by one or more processors, the resource manager programmed to;
receive at least one database query directed to the database data by a computerized query source;
identify one or more of the files within the database data that must be processed to respond to the at least one database query;
divide processing of the one or more of the files into multiple discrete tasks; and
assign different tasks of the multiple discrete tasks to different nodes within a plurality of nodes of an execution platform;
wherein the resource manager is independent of and remote from the plurality of nodes of the execution platform;
the execution platform executing the multiple discrete tasks, wherein the execution platform comprises a plurality of virtual warehouses, and each virtual warehouse of the plurality of virtual warehouses includes at least one node of the plurality of nodes, and wherein each node of the plurality of nodes is independent of and remote from the computerized query source and from the plurality of shared storage devices and comprises:
at least one processor executing one or more tasks of the multiple discreet tasks assigned thereto by the resource manager; and
at least one local cache caching at least a portion of the database data.

2. The system of claim 1 , wherein the at least one database query is in the form of one or more SQL statements.

3. The system of claim 1 , wherein at least one task of the multiple discrete tasks comprises one or more reads from the plurality of shared storage devices.

4. The system of claim 1 , wherein at least one task of the multiple discrete tasks comprises one or more writes to the plurality of shared storage devices.

5. The system of claim 1 , wherein the resource manager is further programmed to allocate access to the plurality of shared storage devices for a predetermined time period.

6. The system of claim 1 , wherein the resource manager is further programmed to identify at least one storage device of the plurality of shared storage devices that is associated with database data identified in a received SQL statement.

7. The system of claim 1 , wherein the at least one local cache of each node of the plurality of nodes is configured to store database data from any device of the plurality of shared storage devices.

8. The system of claim 7 , wherein each node of the plurality of nodes is connected to:
read database data from any device of the plurality of shared storage devices; and
write database data to any device of the plurality of shared storage devices.

9. The system of claim 7 , wherein each node of the plurality of nodes is configured to read a needed portion of the database data from the at least one local cache corresponding thereto rather than from the plurality of shared storage devices whenever the needed portion of the database data is cached at the at least one local cache.

10. A method comprising:
deploying an interface system between a plurality of shared storage devices collectively storing database data and at least one computerized query source, wherein:
the interface system comprises a resource manager and an execution platform;
the resource manager comprises a software program stored in memory and executed by one or more processors connected to the execution platform;
the execution platform is independent of and remote from the plurality of shared storage devices;
the execution platform is independent of and remote from the at least one computerized query source;
the execution platform comprises a plurality of nodes, wherein the plurality of nodes are organized in a plurality of virtual warehouses such that each virtual warehouse of the plurality of virtual warehouses comprises multiple nodes of the plurality of nodes, each node thereof comprising at least one processor and at least one local cache caching at least a portion of the database data; and
the resource manager is independent of and remote from the plurality of nodes of the execution platform;
receiving, by the resource manager, at least one database query directed to the database data by the at least one computerized query source;
determining a plurality of tasks that must be completed to respond to the at least one database query;
distributing, by the resource manager, the plurality of tasks among the plurality of nodes for processing, wherein each node of the plurality of nodes is assigned at least one task of the plurality of tasks;
providing, by the execution platform to the resource manager, results obtained from processing of the plurality of tasks; and
using, by the resource manager, the results to respond to the at least one database query.

11. The method of claim 10 , wherein the at least one database query comprises one or more SQL statements.

12. The method of claim 10 , further comprising processing, by each node of the plurality of nodes, the at least one task assigned thereto by:
determining whether a portion of the database data necessary to process the at least one task is cached at the at least one local cache corresponding thereto; and
when it is determined that the portion of the database data is cached on the at least one local cache, retrieving the portion of the database data from the at least one local cache; and
when it is determined that the portion of the database data is not cached on the at least one local cache, retrieving the portion of the database data from the plurality of shared storage devices and caching the portion of the database data at the at least one local cache.

13. A system comprising:
means for storing database data;
means for receiving at least one database query directed to the database data by at least one computerized query source;
means for determining which files within the database data must be processed to respond to the at least one database query;
means for dividing processing of the files into multiple discrete tasks; means for assigning different tasks of the multiple discrete tasks to different nodes within a plurality of nodes of an execution platform, wherein the plurality of nodes are organized as a plurality of virtual warehouses, each virtual warehouse of the plurality virtual warehouses comprises more than one node of the plurality of nodes; and
the execution platform executing the multiple discrete tasks, wherein each node of the plurality of nodes is independent of and remote from the at least one database query source and from the means for storing database data and comprises;
at least one processor executing one or more tasks of the multiple discreet tasks assigned thereto by the means for assigning; and
at least one local cache caching at least a portion of the database data, wherein the means for receiving the at least one database query is independent of and remote from the plurality of nodes of the execution platform.

Assignees

• Snowflake Computing Inc
• Snowflake Inc

Inventors

• Dageville Benoit
• Cruanes Thierry
• Zukowski Marcin

Classifications

• G06F16/128

  Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion (error detection or correction of the data by redundancy in operations or in hardware G06F11/14, G06F11/16) · CPC title

• G06F9/5016 Primary

  the resource being the memory · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F16/148

  File search processing · CPC title

• G06F16/211

  Schema design and management · CPC title