Method and system of network transfer adaptive optimization in large-scale parallel computing system

Patent metadata

Field	Value
Publication number	US-9369350-B2
Application number	US-201213685111-A
Country	US
Kind code	B2
Filing date	Nov 26, 2012
Priority date	Dec 1, 2011
Publication date	Jun 14, 2016
Grant date	Jun 14, 2016

Abstract

A method and system for performing network transfer adaptive optimization in a large-scale parallel computing system. The method of network transfer adaptive optimization includes forming a message to be transferred through the network based on obtained information related to a task executed by computing nodes, wherein the message includes identification information of the computing nodes to perform data transfer and corresponding sink nodes, and an amount of data the computing nodes to transfer to the corresponding sink nodes; transferring the message to a network layer; and forming a new data transfer pattern for data transfer between the computing nodes and the corresponding sink nodes in accordance with the received message.

First claim

What is claimed is:

1. A method of network transfer adaptive optimization in a large-scale parallel computing system, comprising:
forming a message to be transferred through the network based on obtained information related to a task executed by computing nodes, wherein the message includes identification information of the computing nodes to perform data transfer and corresponding sink nodes, and an amount of data said computing nodes to transfer to said corresponding sink nodes;
transferring said message to a network layer; and
forming a new data transfer pattern for data transfer between said computing nodes and said corresponding sink nodes in accordance with a received message,
wherein said forming a message to be transferred through the network includes formatting the obtained information to form a message in a predetermined format, and
wherein said predetermined format is:
<total number of triplets>{one or more <computing node IDs, sink node IDs, size of amount of data>}, wherein the total number of triplets indicates the number of triplets <computing node ID, sink node ID, size of amount of data>in the message.

2. The method of network transfer adaptive optimization according to claim 1 , wherein said forming a new data transfer pattern for data transfer between said computing nodes and said corresponding sink nodes includes:
determining whether locations of said sink nodes are needed to be adjusted in accordance with network information of the network layer and the received message; and
if it is determined that the locations of said sink nodes are needed to be adjusted, replacing the sink nodes in said message with re-selected sink nodes.

3. The method of network transfer adaptive optimization according to claim 2 , wherein said replacing the sink nodes in said message with the re-selected sink nodes includes:
replacing the sink nodes in said message in turn from a most preferred selected sink node, and when the amount of data of the replaced sink nodes reaches a capacity of the most preferred selected sink node, continuing to replace the sink nodes in said message with a secondarily preferred selected sink node.

4. The method of network transfer adaptive optimization according to claim 2 , wherein said determining whether the locations of said sink nodes are needed to be adjusted includes:
calculating a sum of distances from each location at which a sink node may be located in said large-scale parallel computing system to each of all the computing nodes;
selecting a desired number of locations in turn from a possible location with a smallest distance as the locations of said sink nodes; and
if the selected locations are different from the locations indicated by identification information of the sink nodes included in said message, determining that the adjustment is required.

5. The method of network transfer adaptive optimization according to claim 4 , wherein said sum of distances is a weighted sum after weighting the distances using the amount of data.

6. The method of network transfer adaptive optimization according to claim 3 , wherein said message further includes a task attribute of said task, said task attribute indicating whether multicast is required by said task, and
wherein said method further comprises:
if said message indicates that multicast is required by said task, setting a multicast group, by a network engine, directed to each computing node and all the sink nodes.

7. The method of network transfer adaptive optimization according to claim 1 , wherein said forming a new data transfer pattern for data transfer between said computing nodes and said corresponding sink nodes includes:
re-selecting a data transfer path between said computing nodes and said corresponding sink nodes in accordance with network information of said network layer and the received message.

8. The method of network transfer adaptive optimization according to claim 7 , wherein said re-selecting a data transfer path between said computing nodes and said corresponding sink nodes includes:
marking each link between said computing nodes and said corresponding sink nodes in said message using said amount of data as a load;
calculating a weight value of the load for each link based on acquired network information; and
selecting the data transfer path between said computing nodes and said corresponding sink nodes based on the weight value of the load.

9. The method of network transfer adaptive optimization according to claim 8 , wherein the weight value of the load is calculated from a sum of a current link load indicated by the acquired network information and a ratio of the load and bandwidth as marked for the link.

10. The method of network transfer adaptive optimization according to claim 8 , further comprising:
if said message indicates that multicast is required by said task, creating a multicast tree from each computing node indicated in said message to all the sink nodes; and
for the multicast tree, executing re-selection of said data transfer path.

11. The method of network transfer adaptive optimization according to claim 8 , wherein said selecting the data transfer path between said computing nodes and said corresponding sink nodes is further based on a shortest path between said computing nodes and said corresponding sink nodes.

12. The method of network transfer adaptive optimization according to claim 1 , further comprising:
deciding whether said task is a shuffle critical task; and
if said task is not the shuffle critical task, ignoring said task thereby to ignore information related to said task.

13. The method of network transfer adaptive optimization according to claim 1 , wherein said forming a new data transfer pattern includes:
filtering the received message to remove the computing nodes with a smaller amount of data and the corresponding sink nodes from said message, and
wherein said method of network transfer adaptive optimization further comprises:
feeding back information related to the formed new data transfer pattern to said computing nodes.

14. An apparatus of network transfer adaptive optimization in a large-scale parallel computing system, comprising:
an application engine configured to form a message to be transferred through a network based on obtained information related to a task executed by computing nodes, wherein the message includes identification information of the computing nodes to perform data transfer and corresponding sink nodes, and an amount of data said computing nodes to transfer to said corresponding sink nodes; and
a network engine configured to form a new data transfer pattern for the data transfer between said computing nodes and said corresponding sink nodes in accordance with a message received from the application engine,
wherein said application engine includes a network interface configured to receive said information; and a formatter configured to format the obtained said information to form a message in a predefined format, and
wherein said predefined format is:
<total number of triplets>{one or more <computing node IDs, sink node IDs, size of amount of data>}, wherein the total number of triplets indicates the number of triplets <computing node ID, sink node ID, size of amount of data>in the message.

15. The apparatus of network transfer adaptive optimization according to claim 14 , wherein said network engine includes:
an adjuster configured to determine whether locations of said sink nodes are needed to be adjusted in accordance with network information of a network layer of the network and in accordance with the message recei

Assignees

• IBM

Inventors

• Li Yan
• Lin Hai Bo
• Zhang Yue
• Zheng Kai

Classifications

• G06F11/3065

  Monitoring arrangements determined by the means or processing involved in reporting the monitored data (error or fault reporting or logging G06F11/0766) · CPC title

• H04L41/26 Primary

  using dedicated tools for LAN [Local Area Network] management · CPC title

• G06F11/3017

  where the computing system is implementing multitasking (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50) · CPC title

• H04L45/16

  Multipoint routing · CPC title

• G06F11/3433

  for load management (allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029) · CPC title