Optimized resource management for map/reduce computing

Patent metadata

Field	Value
Publication number	US-9367359-B2
Application number	US-82824510-A
Country	US
Kind code	B2
Filing date	Jun 30, 2010
Priority date	Jun 30, 2010
Publication date	Jun 14, 2016
Grant date	Jun 14, 2016

Abstract

Embodiments of the present invention include a method for resource optimization of map/reduce computing in a computing cluster. The method can include receiving a computational problem for processing in a map/reduce module, subdividing the computational problem into a set of sub-problems and mapping a selection of the sub-problems in the set to respective nodes in a computing cluster, for example a cloud computing cluster, computing for a subset of the nodes in the computing cluster a required resource capacity of the subset of the nodes to process a mapped one of the sub-problems and an existing capacity of the subset of the nodes, and augmenting the existing capacity to an augmented capacity when the required resource capacity exceeds the existing capacity, and when a cost of augmenting the existing capacity to the augmented capacity does not exceed a penalty for breaching a service level agreement (SLA) for the subset of the nodes.

First claim

We claim:

1. A map/reduce data processing system comprising:
a computer with at least one processor and a memory communicatively coupled to a computing cluster of nodes over a computer communications network and hosting execution of a map/reduce module;
a service level agreement (SLA) stored in a fixed storage coupled to the computer; and,
optimized resource management logic coupled to the map/reduce module, the optimized resource management logic comprising program code enabled to compute for a subset of nodes in the computing cluster of nodes a required resource capacity of the subset of nodes to process a mapped sub-problem of a computational problem submitted to the map/reduce module and also an existing capacity of the subset of nodes, to determine whether the required resource capacity of the subset of nodes exceeds the existing capacity of the subset of nodes, upon determining that the required resource capacity of the subset of the nodes exceeds the existing capacity of the subset of the nodes, both determine a cost of augmenting the existing capacity to an augmented capacity to elevate the subset of the nodes to the required resource capacity and also determine a penalty for breaching the SLA, and to augment the existing capacity of the subset of nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity upon both determining that the required resource capacity exceeds the existing capacity and also upon determining that the cost of augmenting the existing capacity to the augmented capacity to elevate the subset of the nodes to the required resource capacity does not exceed the penalty for breaching the SLA.

2. The system of claim 1 , wherein the computing cluster of nodes is a cloud computing cluster of nodes.

3. The system of claim 1 , wherein the subset of nodes is a server in the computing cluster of nodes.

4. A computer program product for optimized resource management of map/reduce computing in a computing cluster, the computer program product comprising:
a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code for receiving a computational problem for processing in a map/reduce module executing in a memory by at least one processor of a computer;
computer readable program code for subdividing the computational problem into a set of sub-problems and mapping a selection of the sub-problems in the set of sub-problems to respective nodes in the computing cluster;
computer readable program code for computing for a subset of the nodes in the computing cluster a required resource capacity of the subset of the nodes to process a mapped one of the sub-problems and also an existing capacity of the subset of the nodes;
computer readable program code for determining whether the required resource capacity of the subset of the nodes exceeds the existing capacity of the subset of the nodes;
computer readable program code for, upon determining that the required resource capacity of the subset of the nodes exceeds the existing capacity of the subset of the nodes, both determining a cost of augmenting the existing capacity to an augmented capacity to elevate the subset of the nodes to the required resource capacity and also determining a penalty for breaching a service level agreement (SLA) for the computing cluster; and,
computer readable program code for augmenting the existing capacity of the subset of the nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity upon both determining that the required resource capacity exceeds the existing capacity, and also upon determining that the cost of augmenting the existing capacity to the augmented capacity to elevate the subset of the nodes to the required resource capacity does not exceed a penalty for breaching the SLA for the computing cluster.

5. The computer program product of claim 4 , wherein the computing cluster is a cloud computing cluster.

6. The computer program product of claim 4 , wherein the computer readable program code for augmenting the existing capacity of the subset of the nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises computer readable program code for provisioning additional servers in the computing cluster for the subset of the nodes.

7. The computer program product of claim 4 , wherein the computer readable program code for augmenting the existing capacity of the subset of the nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises computer readable program code for provisioning additional memory in the subset of the nodes.

8. The computer program product of claim 4 , wherein the computer readable program code for augmenting the existing capacity of the subset of the nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises computer readable program code for provisioning additional processors in the subset of the nodes.

9. The computer program product of claim 5 , wherein the computer readable program code for augmenting the existing capacity of the subset of the nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises computer readable program code for augmenting the existing capacity of the subset of the nodes to the augmented capacity when the required resource capacity exceeds by a first threshold value of the existing capacity, and also when the cost of augmenting the existing capacity to the augmented capacity does not exceed beyond a second threshold value of a penalty for breaching a service level agreement (SLA) for the cloud computing cluster.

10. The system of claim 1 , wherein the program code enabled to augment the existing capacity of the subset of nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises program code to provision additional servers in the computing cluster of nodes for the subset of the nodes.

11. The system of claim 1 , wherein the program code enabled to augment the existing capacity of the subset of nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises program code to provision additional memory in the subset of the nodes.

12. The system of claim 1 , wherein the program code enabled to augment the existing capacity of the subset of nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises program code to provision additional processors in the subset of the nodes.

13. The system of claim 2 , wherein the program code enabled to augment the existing capacity of the subset of nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises program code to augment the existing capacity of the subset of the nodes to the augmented capacity when the required resource capacity exceeds by a first threshold value of the existing capacity, and also when the cost of augmenting the existing capacity to the augmented capacity does not exceed beyond a second threshold value of a penalty for breaching a service level agreement (SLA) for the cloud computing cluster.

14. The system of claim 1 , wherein the program code enabled to augment the existing capacity of the subset of nodes to the augmented capacity to elevate the subset of the nodes to the required resource capacity comprises program code to provision additional ban

Assignees

• Doyle Ronald P
• Kaminsky David L
• IBM

Inventors

• Doyle Ronald P
• Kaminsky David L

Classifications

• H04L41/0813

  characterised by the conditions triggering a change of settings · CPC title

• G06F9/5066 Primary

  Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451) · CPC title

• H04L41/0896

  Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities (flow or congestion control using dynamic resource allocation, e.g. in-call renegotiation, H04L47/76) · CPC title