Distributed processing system and distributed processing method

Patent metadata

Field	Value
Publication number	US-11240296-B2
Application number	US-201917287063-A
Country	US
Kind code	B2
Filing date	Oct 7, 2019
Priority date	Oct 22, 2018
Publication date	Feb 1, 2022
Grant date	Feb 1, 2022

Abstract

A first distributed processing node transmits distributed data to a second distributed processing node as intermediate consolidated data. A third distributed processing node generates intermediate consolidated data after update from received intermediate consolidated data and distributed data, and transmits the intermediate consolidated data to a fourth distributed processing node. The first distributed processing node transmits the received intermediate consolidated data to fifth distributed processing node as consolidated data. The third distributed processing node transmits the received consolidated data to a sixth distributed processing node. When an aggregation communication time period required by each distributed processing node to consolidate the distributed data or an aggregation dispatch communication time period being a total time period of the aggregation communication time period and a time period required by each distributed processing node to dispatch the consolidated data exceeds a predetermined time period, the first distributed processing node issues a warning.

First claim

The invention claimed is:

1. A distributed processing system comprising:
N distributed processing nodes disposed in a ring shape and connected to adjacent nodes via a communication path, wherein N is an integer of 2 or greater, wherein:
an n-th (n=1, . . . , N) distributed processing node includes a first communication port configured to perform simultaneous bidirectional communication with an n + -th distributed processing node, wherein n + =n+1 provided that n + =1 if n=N, and a second communication port configured to perform simultaneous bidirectional communication with an n − -th distributed processing node, wherein n − =n−1 provided that n − =N if n=1;
each of the distributed processing nodes is configured to generate distributed data for each of M weights w [m] of a neural network of a learning target, wherein M is an integer of 2 or greater and m=1, . . . , M;
out of the N distributed processing nodes, a first distributed processing node specified in advance is configured to use the distributed data generated in the first distributed processing node as first consolidated data, packetize the first consolidated data in order of numbers m of the weights w [m], and transmit the first consolidated data from the first communication port of the first distributed processing node to a second distributed processing node;
out of the N distributed processing nodes, a k-th distributed processing node except the first distributed processing node is configured to generate updated first consolidated data by calculating a sum of the first consolidated data received via the second communication port of the k-th distributed processing node from a (k−1)-th distributed processing node and the distributed data generated in the k-th distributed processing node for each corresponding one of the weights w [m], wherein k=2, . . . , N, packetize the first consolidated data in the order of the numbers m, and transmit the first consolidated data from the first communication port of the k-th distributed processing node to a k + -th distributed processing node, wherein k + =k+1 provided that k + =1 if k=N;
the first distributed processing node is configured to use the first consolidated data received via the second communication port of the first distributed processing node from an N-th distributed processing node as second consolidated data, packetize the second consolidated data in the order of the numbers m, and transmit the second consolidated data from the second communication port of the first distributed processing node to the N-th distributed processing node;
the k-th distributed processing node is configured to packetize the second consolidated data received via the first communication port of the k-th distributed processing node from the k + -th distributed processing node in the order of the numbers m, and transmit the second consolidated data from the second communication port of the k-th distributed processing node to the (k−1)-th distributed processing node;
the first distributed processing node is configured to receive the second consolidated data from the second distributed processing node via the first communication port of the first distributed processing node;
each distributed processing node is configured to update the weights w [m] of the neural network, based on the second consolidated data; and
when an aggregation communication time period required by each of the distributed processing nodes to consolidate the distributed data or an aggregation dispatch communication time period being a total time period of the aggregation communication time period and a time period required by each of the distributed processing nodes to dispatch the second consolidated data exceeds a predetermined maximum consolidation delay time period, the first distributed processing node is configured to issue a warning indicating a consolidation delay anomaly.

2. The distributed processing system according to claim 1 , wherein each of the N distributed processing nodes includes:
an in-node consolidation processor configured to generate the distributed data;
a first transmitter configured to, when a respective one of the distributed processing nodes functions as the first distributed processing node, packetize the first consolidated data in the order of the numbers m of the weights w [m] and transmit the first consolidated data from the first communication port of the respective one of the distributed processing nodes to the second distributed processing node, and configured to, when the respective one of the distributed processing nodes functions as the k-th distributed processing node, packetize the updated first consolidated data in the order of the numbers m and transmit the updated first consolidated data from the first communication port of the respective one of the distributed processing nodes to the k + -th distributed processing node;
a first receiver configured to acquire the first consolidated data from a packet received from the second communication port of the respective one of the distributed processing nodes;
a second transmitter configured to, when the respective one of the distributed processing nodes functions as the first distributed processing node, packetize the second consolidated data in the order of the numbers m and transmit the second consolidated data from the second communication port of the respective one of the distributed processing nodes to the N-th distributed processing node, and configured to, when the respective one of the distributed processing nodes functions as the k-th distributed processing node, packetize the received second consolidated data in the order of the numbers m and transmit the received second consolidated data from the second communication port of the respective one of the distributed processing nodes to the (k−1)-th distributed processing node;
a second receiver configured to acquire the second consolidated data from a packet received from the first communication port of the respective one of the distributed processing nodes;
a consolidated data generator configured to generate the updated first consolidated data when the respective one of the distributed processing nodes functions as the k-th distributed processing node;
a weight updating processor configured to update the weights w [m] of the neural network, based on the received second consolidated data;
a timer configured to, when the respective one of the distributed processing nodes functions as the first distributed processing node, measure a time period from a time point when the first consolidated data is transmitted to the second distributed processing node to a time point when the first consolidated data is received from the N-th distributed processing node as the aggregation communication time period, and a time period from a time point when the first consolidated data is transmitted to the second distributed processing node to a time point when the second consolidated data is received from the second distributed processing node as the aggregation dispatch communication time period; and
a warning issuer configured to, when the respective one of the distributed processing nodes functions as the first distributed processing node, issue a warning indicating a consolidation delay anomaly when the aggregation communication time period or the aggregation dispatch communication time period exceeds the maximum consolidation delay time period.

3. The distributed processing system according to claim 2 , wherein:
the first distributed processing node is configured to regularly generate a management packet including a consolidation start confirmation flag indicating whether or not preparation for consolidating the distributed data is completed, before the first consolidated data is transmitted from the first communication port of the first distributed processing node, and trans

Assignees

• Nippon Telegraph & Telephone

Inventors

• Kawai Kenji
• Kato Junichi
• Ngo Huycu
• Arikawa Yuki
• Ito Tsuyoshi
• Sakamoto Takeshi

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• H04L69/40

  for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection (management of faults, events, alarms or notifications in data switching networks H04L41/06) · CPC title

• G06F15/173

  using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake · CPC title

• H04L43/022

  by sampling · CPC title