Multi-cluster warehouse

Patent metadata

Field	Value
Publication number	US-11615117-B2
Application number	US-201715582071-A
Country	US
Kind code	B2
Filing date	Apr 28, 2017
Priority date	Apr 28, 2016
Publication date	Mar 28, 2023
Grant date	Mar 28, 2023

Abstract

A method for a multi-cluster warehouse includes allocating a plurality of compute clusters as part of a virtual warehouse. The compute clusters are used to access and perform queries against one or more databases in one or more cloud storage resources. The method includes providing queries for the virtual warehouse to each of the plurality of compute clusters. Each of the plurality of compute clusters of the virtual warehouse receives a plurality of queries so that the computing load is spread across the different clusters. The method also includes dynamically adding compute clusters to and removing compute clusters from the virtual warehouse as needed based on a workload of the plurality of compute clusters.

First claim

The invention claimed is:

1. A system for implementing a fault-tolerant data warehouse using availability zones, comprising:
a means for allocating a plurality of compute clusters on an execution platform as part of a virtual warehouse for accessing and performing queries against one or more databases in one or more cloud storage resources located on a storage platform separate from the execution platform, wherein the plurality of compute clusters is allocated separately from the one or more cloud storage resources;
a means for routing queries directed to data within the one or more cloud storage resources to each of the plurality of compute clusters, the plurality of compute clusters comprising at least two processing units in different availability zones, an availability zone comprising one or more data centers, each data center comprising redundant power, networking, and connectivity, wherein a plurality of queries is provided to each of the plurality of compute clusters of the virtual warehouse, the plurality of queries having a common session identifier with a query previously provided to the compute cluster, the compute cluster further determined to be caching a data segment usable by the plurality of queries, and each of the plurality of compute clusters of the virtual warehouse comprise a processor and a cache memory to cache data stored in the one or more cloud storage resources; and
a means for dynamically adding compute clusters to or removing compute clusters from the virtual warehouse based on a workload of the plurality of compute clusters, the workload using at least in part on a comparison of a runtime computed degree of concurrency on each of the plurality of compute clusters and a targeted degree of concurrency inputted by a customer, the runtime computed degree of concurrency is computed using a number of queries running at an input degree of concurrency, and wherein the means for adding or removing the compute clusters scales up and down a number of compute clusters without increasing or decreasing the one or more cloud storage resources.

2. The system of claim 1 , wherein the means for dynamically adding compute clusters to and removing compute clusters from the virtual warehouse based on the workload comprises:
a means for determining whether a query can be processed while meeting a performance metric for the query; and
a means for triggering startup of a new compute cluster in response to determining that the query in combination with a current workload does not allow one or more currently allocated compute clusters to meet the performance metric.

3. The system of claim 1 , wherein the means for dynamically adding compute clusters to and removing compute clusters from the virtual warehouse based on the workload comprises:
a means for determining whether a current workload is serviceable by one fewer than the plurality of compute clusters while meeting a performance metric; and
a means for decommissioning at least one compute cluster of the plurality of compute clusters in response to determining that the workload is serviceable by one fewer than the plurality of compute clusters.

4. The system of claim 1 , wherein the means for routing queries for the virtual warehouse to each of the plurality of compute clusters comprises one or more of:
a means for routing queries based on a session from which a query originated;
a means for routing queries based on cluster availability; or
a means for routing queries based on availability of cluster resources to execute a query.

5. The system of claim 1 , wherein the means for adding compute clusters to or removing compute clusters from the virtual warehouse comprise means for dynamically adding compute clusters to or removing compute clusters from the virtual warehouse as needed.

6. A method for implementing a fault-tolerant data warehouse using availability zones, comprising:
allocating a plurality of compute clusters on an execution platform as part of a virtual warehouse for accessing and performing queries against one or more databases in one or more cloud storage resources located on a storage platform separate from the execution platform, wherein the plurality of compute clusters is allocated separately from the one or more cloud storage resources;
routing queries directed to data within the one or more cloud storage resources to each of the plurality of compute clusters, the plurality of compute clusters comprising at least two processing units in different availability zones, an availability zone comprising one or more data centers, each data center comprising redundant power, networking, and connectivity, wherein a plurality of queries is provided to each of the plurality of compute clusters of the virtual warehouse, the plurality of queries having a common session identifier with a query previously provided to the compute cluster, the compute cluster further determined to be caching a data segment usable by the plurality of queries, and each of the plurality of compute clusters of the virtual warehouse comprise a processor and a cache memory to cache data stored in the one or more cloud storage resources; and
dynamically adding, by one or more processors, compute clusters to or removing compute clusters from the virtual warehouse based on a workload of the plurality of compute clusters, the workload using at least in part on a comparison of a runtime computed degree of concurrency on each of the plurality of compute clusters and a targeted degree of concurrency inputted by a customer, the runtime computed degree of concurrency is computed using a number of queries running at an input degree of concurrency, and wherein the adding or removing the compute clusters does not increase or decrease the one or more cloud storage resources.

7. The method of claim 6 , further comprising determining the workload for the plurality of compute clusters, wherein determining the workload comprises determining availability of one or more of:
processor resources for each of the plurality of compute clusters; and
memory resources for each of the plurality of compute clusters.

8. The method of claim 6 , wherein adding compute clusters to the virtual warehouse based on the workload comprises:
determining whether the queries can be processed while meeting a performance metric for the queries; and
triggering startup of a new compute cluster in response to determining that the query in combination with a current workload does not allow one or more currently allocated compute clusters to meet the performance metric.

9. The method of claim 8 , wherein the method comprises determining whether the queries can be processed for each query directed to a compute cluster such that the performance metric is met for each query.

10. The method of claim 8 , wherein the performance metric comprises a service level agreement accepted by a customer.

11. The method of claim 8 , wherein the performance metric comprises a maximum time period that the query will be queued.

12. The method of claim 6 , wherein adding compute clusters comprises adding compute clusters up to a predetermined maximum number of compute clusters.

13. The method of claim 6 , wherein removing compute clusters comprises removing compute clusters down to a predetermined minimum number of compute clusters.

14. The method of claim 6 , wherein removing compute clusters based on the workload comprises:
determining whether a current workload is serviceable by one fewer than the plurality of compute clusters while meeting a performance metric; and
decommissioning at least one compute cluster of the plurality of compute clusters in re

Assignees

• Snowflake Inc

Inventors

• Funke Florian Andreas
• Povinec Peter
• Cruanes Thierry
• Dageville Benoit

Classifications

• H04L41/0896

  Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities (flow or congestion control using dynamic resource allocation, e.g. in-call renegotiation, H04L47/76) · CPC title

• G06F16/283 Primary

  Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP · CPC title

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• G06F16/2455

  Query execution · CPC title

• H04L41/5025

  by proactively reacting to service quality change, e.g. by reconfiguration after service quality degradation or upgrade · CPC title