Flexible computing

Patent metadata

Field	Value
Publication number	US-11513859-B2
Application number	US-202117342713-A
Country	US
Kind code	B2
Filing date	Jun 9, 2021
Priority date	May 14, 2020
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

Embodiments of the present disclosure may provide dynamic and fair assignment techniques for allocating resources on a demand basis. Assignment control may be separated into at least two components: a local component and a global component. Each component may have an active dialog with each other; the dialog may include two aspects: 1) a demand for computing resources, and 2) a total allowed number of computing resources. The global component may allocate resources from a pool of resources to different local components, and the local components in turn may assign their allocated resources to local competing requests. The allocation may also be throttled or limited at various levels.

First claim

What is claimed is:

1. A method comprising:
receiving a demand of computing resources from a plurality of compute service managers in a data warehouse, each of the compute service managers being defined for a different type of data processing services, and the demand being read from a metadata database in the data warehouse system;
calculating a first total demand of computing resources for the data warehouse system based at least on the received demand;
based on the first total demand, requesting a first number of computing resources from a pool of computing resources;
receiving a first distribution of computing resources;
allocating a first portion of the first distribution of the computing resources to a first compute service manager of a plurality of compute service managers, the first compute service manager being defined by a first data processing service type;
allocating a second portion of the first distribution of the computing resources to a second compute service manager of a plurality of compute service managers, the second compute service manager being defined by a second data processing service type;
receiving an updated demand for computing resources from the plurality of compute service managers;
based on the updated demand, receiving a second distribution of computing resources;
based on the second distribution, reducing allocation of the first portion of the computing resources to the first compute service manage;
transmitting an instruction to the first compute service manager to release one of the computing resources in the first portion of the computing resources, the instruction being written to the metadata database;
receiving a notification from the first compute service manager regarding the release of one of the computing resources in the first portion of the computing resources, the notification being read from the metadata database; and
allocating the released computing resource to the second compute service manager.

2. The method of claim 1 , further comprising:
receiving per source demand information from the plurality of compute service managers, wherein the first distribution of computing resources is based on the per source demand information.

3. The method of claim 1 , wherein the updated demand is based on historical data.

4. The method of claim 1 , wherein allocating the first and second portions is based on a fair distribution algorithm.

5. The method of claim 4 , wherein the fair distribution algorithm includes a Max-Min fairness algorithm.

6. The method of claim 1 , wherein the updated demand for computing resources is based on consolidated query information received by the plurality of compute service managers.

7. The method of claim 1 , wherein the computing resources comprises virtual computing resources.

8. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
receiving a demand of computing resources from a plurality of compute service managers in a data warehouse, each of the compute service managers being defined for a different type of data processing services, and the demand being read from a metadata database in the data warehouse system;
calculating a first total demand of computing resources for the data warehouse system based at least on the received demand;
based on the first total demand, requesting a first number of computing resources from a pool of computing resources;
receiving a first distribution of computing resources;
allocating a first portion of the first distribution of the computing resources to a first compute service manager of a plurality of compute service managers, the first compute service manager being defined by a first data processing service type;
allocating a second portion of the first distribution of the computing resources to a second compute service manager of a plurality of compute service managers, the second compute service manager being defined by a second data processing service type;
receiving an updated demand for computing resources from the plurality of compute service managers;
based on the updated demand, receiving a second distribution of computing resources;
based on the second distribution, reducing allocation of the first portion of the computing resources to the first compute service manage;
transmitting an instruction to the first compute service manager to release one of the computing resources in the first portion of the computing resources, the instruction being written to the metadata database;
receiving a notification from the first compute service manager regarding the release of one of the computing resources in the first portion of the computing resources, the notification being read from the metadata database; and
allocating the released computing resource to the second compute service manager.

9. The system of claim 8 , the operations further comprising:
receiving per source demand information from the plurality of compute service managers, wherein the first distribution of computing resources is based on the per source demand information.

10. The system of claim 8 , wherein the updated demand is based on historical data.

11. The system of claim 8 , wherein allocating the first and second portions is based on a fair distribution algorithm.

12. The system of claim 11 , wherein the fair distribution algorithm includes a Max-Min fairness algorithm.

13. The system of claim 8 , wherein the updated demand for computing resources is based on consolidated query information received by the plurality of compute service managers.

14. The system of claim 8 , wherein the f computing resources comprises virtual computing resources.

15. A non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:
receiving a demand of computing resources from a plurality of compute service managers in a data warehouse, each of the compute service managers being defined for a different type of data processing services, and the demand being read from a metadata database in the data warehouse system;
calculating a first total demand of computing resources for the data warehouse system based at least on the received demand;
based on the first total demand, requesting a first number of computing resources from a pool of computing resources;
receiving a first distribution of computing resources;
allocating a first portion of the first distribution of the computing resources to a first compute service manager of a plurality of compute service managers, the first compute service manager being defined by a first data processing service type;
allocating a second portion of the first distribution of the computing resources to a second compute service manager of a plurality of compute service managers, the second compute service manager being defined by a second data processing service type;
receiving an updated demand for computing resources from the plurality of compute service managers;
based on the updated demand, receiving a second distribution of computing resources;
based on the second distribution, reducing allocation of the first portion of the computing resources to the first compute service manage;
transmitting an instruction to the first compute service manager to release one of the computing resources in the first portion of the computing resources, the instruction being written to the metadata database;
receiving a notification from the first compute service mana

Assignees

• Snowflake Inc

Inventors

• Cruanes Thierry
• Demura Igor
• Ganesh Varun
• Rajaperumal Prasanna
• Wang Libo
• Yan Jiaqi

Classifications

• H04L67/561

  Adding application-functional data or data for application control, e.g. adding metadata · CPC title

• G06F2209/5011

  Pool · CPC title

• G06F9/547

  Remote procedure calls [RPC]; Web services · CPC title

• G06F9/5022

  Mechanisms to release resources · CPC title

• G06F9/4881 Primary

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title