Hierarchical highly heterogeneous distributed system based deep learning application optimization framework

Patent metadata

Field	Value
Publication number	US-11599789-B2
Application number	US-201816472776-A
Country	US
Kind code	B2
Filing date	Aug 2, 2018
Priority date	Aug 2, 2018
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

The present invention discloses a hierarchical highly heterogeneous distributed system based deep learning application optimization framework and relates to the field of deep learning in the direction of computational science. The hierarchical highly heterogeneous distributed system based deep learning application optimization framework comprises a running preparation stage and a running stage. The running preparation stage is used for performing deep neural network training. The running stage performs task assignment to all kinds of devices in the distributed system and uses a data encryption module to perform privacy protection to user sensitive data. Due to heterogeneous characteristics of a system task of the present invention, on the premise that the overall performance is guaranteed, the system response time is reduced, the user experience is guaranteed, the data encryption module based on the neural network can perform privacy protection to user sensitive data at a lower computing cost and storage cost, and the user data security is guaranteed.

First claim

The invention claimed is:

1. A hierarchical highly heterogeneous distributed system based deep learning application optimization framework, comprising a running preparation stage and a running stage, wherein the running preparation stage is used for performing deep neural network training, and the running stage performs task assignment to all kinds of devices in a distributed system and uses a data encryption module to perform privacy protection to user sensitive data;
wherein the data encryption module is configured that when recognizing a current data is the sensitive data, the data encryption module uses a plurality of former layers of a neural network used in a high-hierarchy computing node to perform forward transmission of the sensitive data; and
wherein, at the running stage, according to a computing node of current task deployment obtained by a task scheduling algorithm based on computing time matching, encrypted data is sent to a designated node for computation, then only a computing result is transmitted to the high-hierarchy computing node, such that original sensitive data is prevented from being transmitted to other nodes.

2. The hierarchical highly heterogeneous distributed system based deep learning application optimization framework according to claim 1 , wherein, in the deep neural network training, deep neural network models having different emphases can be pertinently selected and designed.

3. The hierarchical highly heterogeneous distributed system based deep learning application optimization framework according to claim 2 , wherein a selection of the deep neural network models depends on parameters comprising characteristics of each node in the current hierarchical highly heterogeneous distributed system, including computing ability, power consumption limitation, storage limitation, network state, and support framework.

4. The hierarchical highly heterogeneous distributed system based deep learning application optimization framework according to claim 1 , wherein, in the deep neural network training, a desired neural network is established on computing nodes having enough computing ability after relevant parameters of the deep neural network are determined.

5. The hierarchical highly heterogeneous distributed system based deep learning application optimization framework according to claim 2 , wherein each node of the deep neural network models can differ in network structure, a number of network layers, whether to comprise a data regularization layer, whether to use a convolutional neural network, and whether to use a speed-optimized deep neural network layer.

6. The hierarchical highly heterogeneous distributed system based deep learning application optimization framework according to claim 1 , wherein in the deep natural network training, a training termination condition is referred to a numerical value of a model loss function, an accuracy of a model in a verification data sets, and model training time.

7. The hierarchical highly heterogeneous distributed system based deep learning application optimization framework according to claim 1 , wherein the task scheduling algorithm based on computing time matching computes an optimum matching quantity of terminal nodes and fog nodes according to task completion time of the terminal nodes and the fog nodes to assist in scheduling.

8. The hierarchical highly heterogeneous distributed system based deep learning application optimization framework according to claim 1 , wherein, at the running stage, different task results returned by a computing node are summarized according to heterogeneous characteristics of a system task, which is mainly based on time consumption produced when the task results are returned, a task type of the computing node, and performance of the computing node when the task type is executed.

Assignees

• Univ Shanghai Jiaotong

Inventors

• Ma Ruhui
• Zhang Zongpu
• Song Tao
• Hua Yang
• Guan Haibing

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title