Resource over-subscription

Patent metadata

Field	Value
Publication number	US-9509627-B2
Application number	US-201414501305-A
Country	US
Kind code	B2
Filing date	Sep 30, 2014
Priority date	Dec 12, 2013
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

Embodiments include a method, system, and computer program product for managing workloads in a network. A switch receives data associated with a workload. The received data is tagged with an identifier that associates the data with the workload. The received data is compressed based on determining that second data stored in a buffer of the switch exceeds a threshold. The switch stores the compressed data in the buffer. The compressed data is transmitted to a second network based on a determination that the switch is over-subscribed.

First claim

What is claimed is:

1. A computer implemented method performed by computer system embodied in hardware for managing workloads in a network, wherein the network comprises a data network that implements at least one of a Hadoop network switch or a software-defined network (SDN) enabled switch and an independent management network, the switch performing a method comprising:
receiving, by the switch including a buffer therein, data associated with a workload from a server, wherein the workload is part of a shuffle phase of a map-reduced algorithm;
tagging said data with a unique identifier that associates the data with the workload;
determining that the switch is not over-subscribed when second data stored in the buffer of the switch does not exceed a threshold, and determining that the switch is over-subscribed when the second data stored in the buffer of the switch exceeds the threshold, wherein the threshold is selected based on at least one of: a throughput requirement associated with said network comprises said data network and said independent management network, an anticipated maximum rate of incoming data at said at least one switch, and a capacity of said at least one buffer in said at least one switch;
compressing the received data based on determining that the second data stored in the buffer of the switch exceeds the threshold when the switch is over-subscribed;
storing, by the switch, the compressed data in the buffer; and
transmitting the compressed data and the unique identifier to a controller of said independent management network in response to determining that the switch in the data network is over-subscribed such that spare bandwidth available in said independent management network is leveraged and utilized to transfer the compressed data while the at least one switch is over-subscribed so as to manage the workload between said data network and said independent management network;
receiving, from said controller, said unique identifier and sub-results associated with a processing of said compressed data by said controller, and merging said sub-results with second sub-results identified by said unique identifier to generate overall results;
decompressing said overall results; and
transmitting said overall results to a final destination.

2. The method of claim 1 , wherein the management network monitors the network for errors and monitors the network's performance.

3. The method of claim 1 , further comprising:
processing the compressed data to generate sub-results.

4. The method of claim 3 , further comprising:
merging the sub-results with second sub-results identified by the identifier.

5. The method of claim 1 , wherein the switch is installed in a HADOOP network that includes a Hadoop Distributed File System (HDFS) serving as a storage portion and a Map Reduce serving as a processing portion.

6. The method of claim 1 , wherein the switch is a software-defined network (SDN) enabled switch.

7. A computer program product executable by a computer system embodied in hardware for managing workloads in a network, the computer program product comprising:
a tangible non-transitory computer storage medium readable by a hardware processing circuit and storing instructions for execution by the processing circuit,
wherein the network comprises a data network and an independent management network, the data network including a switch including a buffer, the switch comprising at least one at least one of a Hadoop network switch or a software-defined network (SDN) enabled switch that performs a method comprising:
receiving data associated with a workload from a server, wherein the workload is part of a shuffle phase of a map-reduce algorithm;
tagging said data with a unique identifier that associates the data with the workload;
determining that the switch is not over-subscribed when second data stored in a buffer of the switch does not exceed a threshold, and determining that the switch is over-subscribed when the second data stored in the buffer of the switch exceeds the threshold, the threshold selected based on at least one of: a throughput requirement associated with said network comprises said data network and said independent management network, an anticipated maximum rate of incoming data at said at least one switch, and a capacity of said at least one buffer in said at least one switch;
compressing the received data based on determining that the second data stored in the buffer exceeds the threshold when the switch is over-subscribed;
storing the compressed data in the buffer; and
transmitting the compressed data and the unique identified to said independent management network in response to determining that the switch in the data network is over-subscribed such that available bandwidth available in said independent management network is leveraged and utilized to transfer the compressed data while the at least one switch is over-subscribed so as to manage the workload between said data network and said independent management network;

receiving, from said controller, said unique identifier and sub-results associated with a processing of said compressed data by said controller, and merging said sub-results with second sub-results identified by said unique identifier to generate overall results;
decompressing said overall results; and
transmitting said overall results to a final destination.

8. The computer program product of claim 7 , wherein the management network monitors the network for errors and monitors the network's performance.

9. The computer program product of claim 7 , wherein the method performed by the processing circuit further comprises:
processing the compressed data to generate sub-results.

10. The computer program product of claim 9 , wherein the method performed by the processing circuit further comprises:
merging the sub-results with second sub-results identified by the identifier.

11. The computer program product of claim 7 , wherein the processor is associated with a switch installed in a HADOOP network that includes a Hadoop Distributed File System (HDFS) serving as a storage portion and a Map Reduce serving as a processing portion.

12. The computer program product of claim 7 , wherein the processor is associated with a software-defined network (SDN) enabled switch.

Assignees

• IBM

Inventors

• Decusatis Casimer
• Kashyap Sujatha
• Krishnamurthy Rajaram B

Classifications

• H04L43/08

  Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters · CPC title

• H04L47/70 Primary

  Admission control; Resource allocation · CPC title

• H04L47/30

  in combination with information about buffer occupancy at either end or at transit nodes · CPC title

• H04L43/0823

  Errors, e.g. transmission errors · CPC title

• G06F17/30153

  Physics · mapped topic