Scalable method for optimizing information pathway

Patent metadata

Field	Value
Publication number	US-10169083-B1
Application number	US-201213535760-A
Country	US
Kind code	B1
Filing date	Jun 28, 2012
Priority date	Mar 30, 2012
Publication date	Jan 1, 2019
Grant date	Jan 1, 2019

Abstract

A method, system and process for receiving a request for task execution at a central processing node for the world wide data; wherein the central processing node is connected to sub-processing nodes, dividing the request for task execution to be distributed to a set of the sub-processing nodes; wherein the set of processing nodes manages a portion of the world wide data, transmitting to each of the set of the sub-processing nodes the respective portion of the divided task execution.

First claim

What is claimed is:

1. An apparatus comprising:
a receiving module configured to receive a request for task execution at a central processing node for worldwide data; wherein the central processing node is connected to sub-processing network nodes; wherein the sub-processing network nodes are grouped into clusters; wherein each cluster has a distributed file system mapping out network nodes for each respective cluster; wherein each cluster stores a subset of the worldwide data; and wherein each cluster is enabled to use the network nodes of the cluster to perform parallel processing; wherein the central processing node is communicatively coupled to a global distributed file system that maps over each of the cluster's distributed file systems to enable orchestration between the clusters;
a dividing module configured to divide by a worldwide job tracker the request for task execution into worldwide task trackers to be distributed to sub-processing network nodes of the clusters; wherein the network sub-nodes manages a portion of the worldwide data for each respective cluster; wherein each worldwide task tracker maintains records of sub-activities executed as part of the worldwide job;
a transmitting module configured to transmit to each of the sub-processing network nodes for each respective cluster the respective portion of the divided task execution by assigning each worldwide task tracker corresponding to the respective portion to the respective each cluster; and
a leveraging module configured to generate a graph layout of data pathways, the pathways calculated based upon physical distance between the processing nodes and bandwidth constraints, the leveraging module further configured to distribute task execution based upon the processing power of the processing nodes, graph layout, and the size of data processed by the sub-processing network nodes to reduce data movement between the central processing node and the sub-processing nodes.

2. The apparatus of claim 1 further comprising:
an execution module, configured to coordinate, for each of the sub-processing network nodes for each respective cluster, execution of the respective portion of the divided task execution between the respective cluster of the data network nodes;
a consolidating module configured to consolidate, at each of the sub-processing network nodes for each respective cluster, the coordinated execution of the respective portion of the divided task execution;

a transmitter module configured to transmit the consolidated result of the respective portion of the divided task execution from each of the sub-processing network nodes for each respective cluster to the central processing network nodes; and
a consolidater module configured to consolidate the results of divided task execution from each of the sub-processing network nodes for each respective cluster.

3. The apparatus of claim 2 wherein the sub-processing network nodes for each respective cluster are located on a path between the data and the central processing node.

4. The apparatus of claim 3 further comprising:
a designating configured to designate, for each sub-processing network node for each respective cluster, a second set of sub-processing network nodes between the central processing network node and the data;
a divider module configured to divide the request for task execution to be distributed to the second set of the sub-processing network nodes;
a coordinate module configured to coordinate, for each of the second set of sub-processing network nodes, execution of the respective portion of the divided task execution between a respective cluster; and
consolidating, at each of the second set of sub-processing network nodes, the coordinated execution of the respective portion of the divided task execution.

5. A computer program product comprising:
a non-transitory computer readable medium encoded with computer executable program code, the code configured to enable the execution of:
receiving a request for task execution at a central processing node for worldwide data; wherein the central processing node is connected to sub-processing network nodes; wherein the sub-processing network nodes are grouped into clusters; wherein each cluster has a distributed file system mapping out network nodes for each respective cluster; wherein each cluster stores a subset of the worldwide data; and wherein each cluster is enabled to use the network nodes of the cluster to perform parallel processing; wherein the central processing node is communicatively coupled to a global distributed file system that maps over each of the cluster's distributed file systems to enable orchestration between the clusters;
dividing by a worldwide job tracker the request for task execution into worldwide task trackers to be distributed sub-processing network nodes of the clusters; wherein the sub- processing network nodes manages a portion of the worldwide data for each respective cluster; wherein each worldwide task tracker maintains records of sub-activities executed as part of the worldwide job;
transmitting to each of the sub-processing network nodes for each respective cluster the respective portion of the divided task execution by assigning each worldwide task tracker corresponding to the respective portion to the respective each cluster;
generating a graph layout of data pathways, the pathways calculated based upon physical distance between the processing nodes and bandwidth constraints; and
distributing task execution among processing nodes based upon the graph layout, the processing power of the processing nodes, and the size of data processed by the sub-processing network nodes to reduce the amount and duration of data movement between the central processing node and the sub-processing nodes.

6. The computer program product of claim 5 the code further configured to enable:
for each of the sub-processing network nodes, coordinating execution of the respective portion of the divided task execution between the respective cluster of the data network nodes; consolidating, at each of the subprocessing network nodes for each respective cluster, the coordinated execution of the respective portion of the divided task execution;
transmitting the consolidated result of the respective portion of the divided task execution from each of the sub-processing network nodes for each respective cluster to the central processing nodes; and
consolidating the results of divided task execution from each of the sub-processing network nodes for each respective cluster.

7. The method of claim 6 wherein the sub-processing network nodes for each respective cluster are located on a path between the data and the central processing node.

8. The computer program product of claim 7 the code further configured to enable:
designating, for each sub-processing network node of the sub-processing network nodes for each respective cluster, a second set of sub-processing network nodes between the central processing node and the data;
dividing the request for task execution to be distributed to the second set of the sub-processing network nodes;
for each of the second set of sub-processing network nodes, coordinating execution of the respective portion of the divided task execution between a respective cluster; and
consolidating, at each of the second set of sub-processing network nodes, the coordinated execution of the respective portion of the divided task execution.

9. The computer program product of claim 7 the code further configured to enable: leveraging processing power of the sub-processing network nodes for each respective cluster to minimize data flow to the worldwide data node.

10. The computer program product of claim

Assignees

• Florissi Patricia G S
• Vijendra Sudhir
• Emc Ip Holding Co Llc

Inventors

• Florissi Patricia G S
• Vijendra Sudhir

Classifications

• H04L67/10

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title

• H04W28/00

  Network traffic management; Network resource management · CPC title

• G06F9/4843

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• H04L63/1408

  by monitoring network traffic (monitoring network traffic per se H04L43/00) · CPC title

• G06F2209/509

  Offload · CPC title