Cloud computing data compression for allreduce in deep learning

Patent metadata

Field	Value
Publication number	US-11501160-B2
Application number	US-201916367244-A
Country	US
Kind code	B2
Filing date	Mar 28, 2019
Priority date	Mar 28, 2019
Publication date	Nov 15, 2022
Grant date	Nov 15, 2022

Abstract

In deep learning, and in particular, for data compression for allreduce in deep learning, a gradient may be compressed for synchronization in a data parallel deep neural network training for allreduce by sharing a consensus vector between each node in a plurality of nodes to ensure identical indexing in each of the plurality of nodes prior to performing sparse encoding.

First claim

The invention claimed is:

1. A method for computing compression for allreduce in deep learning in a computing environment by one or more processors comprising:
compressing a gradient for synchronization in data parallel deep neural network training for allreduce by sharing a consensus vector between each node in a plurality of nodes to ensure identical indexing in each of the plurality of nodes prior to performing sparse encoding.

2. The method of claim 1 , further including generating the consensus vector as one bit entry for each number in each index.

3. The method of claim 1 , further including performing a bitwise OR operation on the consensus vector in each index of each node for the allreduce.

4. The method of claim 1 , further including determining the consensus vector for each entry in each index is either above or below a defined threshold prior to performing the sparse encoding.

5. The method of claim 1 , further including maintaining meta data of the consensus vector for each index of each of the plurality of nodes.

6. The method of claim 1 , further including using the consensus vector for reconstructing each index of each of the plurality of nodes.

7. The method of claim 1 , further including:
building a compression mask with the consensus vector by each of the plurality of nodes;
performing a bitwise OR operation on the consensus vector in each of the plurality of nodes for the allreduce;
performing the sparse encoding on the consensus vector;
performing a sum operation on the gradient in each of the plurality of nodes for the allreduce; and
decompress the gradient according to the consensus vector.

8. A system for compression for allreduce in deep learning in a computing environment, comprising:
one or more computers with executable instructions that when executed cause the system to:
compress a gradient for synchronization in data parallel deep neural network training for allreduce by sharing a consensus vector between each node in a plurality of nodes to ensure identical indexing in each of the plurality of nodes prior to performing sparse encoding.

9. The system of claim 8 , wherein the executable instructions further generate the consensus vector as one bit entry for each number in each index.

10. The system of claim 8 , wherein the executable instructions further perform a bitwise OR operation on the consensus vector in each index of each node for the allreduce.

11. The system of claim 8 , wherein the executable instructions further determine the consensus vector for each entry in each index is either above or below a defined threshold prior to performing the sparse encoding.

12. The system of claim 8 , wherein the executable instructions further maintain meta data of the consensus vector for each index of each of the plurality of nodes.

13. The system of claim 8 , wherein the executable instructions further use the consensus vector for reconstructing each index of each of the plurality of nodes.

14. The system of claim 8 , wherein the executable instructions further:
build a compression mask with the consensus vector by each of the plurality of nodes;
perform a bitwise OR operation on the consensus vector in each of the plurality of nodes for the allreduce;
perform the sparse encoding on the consensus vector;
perform a sum operation on the gradient in each of the plurality of nodes for the allreduce; and
decompress the gradient according to the consensus vector.

15. A computer program product for, by a processor, compression for allreduce in deep learning in a computing environment, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:
an executable portion that compresses a gradient for synchronization in data parallel deep neural network training for allreduce by sharing a consensus vector between each node in a plurality of nodes to ensure identical indexing in each of the plurality of nodes prior to performing sparse encoding.

16. The computer program product of claim 15 , further including an executable portion that generates the consensus vector as one bit entry for each number in each index.

17. The computer program product of claim 15 , further including an executable portion that performs a bitwise OR operation on the consensus vector in each index of each node for the allreduce.

18. The computer program product of claim 15 , further including an executable portion that:
determine the consensus vector for each entry in each index is either above or below a defined threshold prior to performing the sparse encoding; and
maintains meta data of the consensus vector for each index of each of the plurality of nodes.

19. The computer program product of claim 15 , further including an executable portion that uses the consensus vector for reconstructing each index of each of the plurality of nodes.

20. The computer program product of claim 15 , further including an executable portion that:
builds a compression mask with the consensus vector by each of the plurality of nodes;
performs a bitwise OR operation on the consensus vector in each of the plurality of nodes for the allreduce;
performs the sparse encoding on the consensus vector;
performs a sum operation on the gradient in each of the plurality of nodes for the allreduce; and
decompresses the gradient according to the consensus vector.

Assignees

• IBM

Inventors

• Cho Minsik
• Zhang Wei
• Finkler Ulrich

Classifications

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• H03M7/6023

  Parallelization · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• H03M7/30

  Compression (speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N); Expansion; Suppression of unnecessary data, e.g. redundancy reduction · CPC title

• H03M7/3066

  by means of a mask or a bit-map · CPC title