Distributed Processing System and Distributed Processing Method

Patent metadata

Field	Value
Publication number	US-2021209443-A1
Application number	US-201916973717-A
Country	US
Kind code	A1
Filing date	May 5, 2019
Priority date	Jun 11, 2018
Publication date	Jul 8, 2021
Grant date	—

Abstract

A first distributed processing node sets, as intermediate aggregated data, distributed data generated by the own node and transmits this data to the distributed processing node having the next number designated in advance. The intermediate distributed processing node excluding the first and last distributed processing nodes calculates, for each of weights corresponding thereto, a sum of the received intermediate aggregated data and distributed data generated by the own node, generates intermediate aggregated data after update, and transmits this data to the distributed processing node having the next number designated in advance. The last distributed processing node calculates, for each of the weights corresponding thereto, a sum of the received intermediate aggregated data and distributed data generated by the own node, generates aggregated data, and transmits this data to the first and intermediate distributed processing nodes. The distributed processing nodes update the weights of a neural network based on this data.

First claim

1 .- 8 . (canceled)

9 . A distributed processing system comprising:
N distributed processing nodes connected to one another via a network, wherein N is an integer equal to or larger than 2, and wherein:
the N distributed processing nodes are configured to generate distributed data for each of M weights w[m] (m=1, . . . , and M) of a learning target neural network, wherein M is an integer equal to or larger than 2,
among the N distributed processing nodes, a predetermined first distributed processing node is configured to set, as first aggregated data, first distributed data generated by itself, packetize the first aggregated data in order of numbers m of the weights w[m], and transmit the first aggregated data to a second distributed processing node having a next number designated in advance,
among the N distributed processing nodes, each of one or more intermediate distributed processing nodes excluding the first distributed processing node and a predetermined last distributed processing node is configured to calculate, for each of the weights w[m] corresponding thereto, a sum of the received first aggregated data and second distributed data generated by itself, generate updated first aggregated data after an update, packetize the updated first aggregated data in the order of the numbers m, and transmit the updated first aggregated data to a following distributed processing node having a next number designated in advance,
among the N distributed processing nodes, the predetermined last distributed processing node is configured to calculate, for each of the weights w[m] corresponding thereto, a sum of the received updated first aggregated data and third distributed data generated by itself, generate second aggregated data, packetize the second aggregated data in the order of the numbers m, and transmit the second aggregated data to the first and the one or more intermediate distributed processing nodes, and
the N distributed processing nodes are configured to update the weights w[m] of the learning target neural network based on the second aggregated data.

10 . The distributed processing system according to claim 9 , wherein each of the distributed processing nodes includes:
an aggregated-data transmitter that, when the distributed processing node is the first distributed processing node, is configured to packetize the first aggregated data in the order of the numbers m and transmit the first aggregated data to the second distributed processing node having the next number designated in advance, when the distributed processing node is one of the one or more intermediate distributed processing nodes, is configured to packetize the updated first aggregated data in the order of the numbers m and transmit the updated first aggregated data to the following distributed processing node having the next number designated in advance, and, when the distributed processing node is the last distributed processing node, is configured to packetize the second aggregated data in the order of the numbers m and transmit the second aggregated data to the first and the one or more intermediate distributed processing nodes; an aggregated-data generator that, when the distributed processing node is one of the intermediate distributed processing nodes, is configured to generate the updated first aggregated data and, when the distributed processing node is the last distributed processing node, is configured to generate the second aggregated data; a receiver that, when the distributed processing node is the first or one of the intermediate distributed processing nodes, is configured to receive the first aggregated data and the second aggregated data and, when the distributed processing node is the last distributed processing node, is configured to receive the first aggregated data; and a weight-update processor configured to update the weights w[m] of the learning target neural network based on the second aggregated data.

11 . A distributed processing system comprising:
K ring nodes in a ring shape and connected to one another by adjacent ring nodes via a communication path, wherein K is an integer equal to or larger than 3; and a distributed-processing controller configured to designate each of the K ring nodes as a distributed processing node or a relay node, wherein:
among the K ring nodes, N ring nodes configured to function as N distributed processing nodes are configured to generate distributed data for each of M weights w[m] (m=1, . . . , and M) of a learning target neural network, wherein N is an integer equal to or larger than 2 and equal to or smaller than K, and wherein M is an integer equal to or larger than 2,
a first ring node configured to function as a first distributed processing node designated in advance among the N distributed processing nodes is configured to set, as first aggregated data, first distributed data generated by itself, packetize the first aggregated data in order of numbers m of the weights w[m], and transmit the first aggregated data to a second distributed processing node having a next number designated in advance,
a second ring node configured to function as an intermediate distributed processing node excluding the first distributed processing node and a last distributed processing node among the N distributed processing nodes is configured to calculate, for each of the weights w[m] corresponding thereto, a sum of the received first aggregated data and second distributed data generated by itself, generate updated first aggregated data after an update, packetize the updated first aggregated data in the order of the numbers m, and transmit the updated first aggregated data to a following distributed processing node having a next number designated in advance,
a third ring node configured to function as the last distributed processing node designated in advance among the N distributed processing nodes is configured to calculate, for each of the weights w[m] corresponding thereto, a sum of the received updated first aggregated data and third distributed data generated by itself, generate second aggregated data, packetize the second aggregated data in the order of the numbers m, and transmit the second aggregated data to a distributed processing node having a preceding number designated in advance,
the second ring node is configured to packetize the received second aggregated data in the order of the numbers m and transmit the second aggregated data to a preceding distributed processing node having a preceding number designated in advance,
among the K ring nodes, the ring nodes configured to function as the relay nodes are configured to transmit the received updated first aggregated data or the received second aggregated data to a distributed processing node at a transfer destination, and
the N distributed processing nodes are configured to update the weights w[m] of the learning target neural network based on the second aggregated data.

12 . The distributed processing system according to claim 11 , wherein each of the ring nodes includes:
an aggregated-data transmitter that, when the ring node is configured to function as the first distributed processing node, is configured to packetize the first aggregated data in the order of the numbers m and transmit the first aggregated data to the second distributed processing node having the next number designated in advance, when the ring node is configured to function as the intermediate distributed processing node, is configured to packetize the updated first aggregated data in the order of the numbers m and transmit the updated first aggregated data to the following distributed processing node having the next number designated in advance, when the ring node is configured to function as the last distributed processing node, is configured to packetize the second aggrega

Assignees

• Nippon Telegraph & Telephone

Inventors

• Kawai Kenji
• Kato Junichi
• Ngo Huycu
• Arikawa Yuki
• Ito Tsuyoshi
• Sakamoto Takeshi

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• G06F9/50

  Allocation of resources, e.g. of the central processing unit [CPU] · CPC title