Domain specific model compression

Patent metadata

Field	Value
Publication number	US-11620435-B2
Application number	US-201916597920-A
Country	US
Kind code	B2
Filing date	Oct 10, 2019
Priority date	Oct 10, 2019
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

Domain specific model compression by providing a weighting parameter for a candidate operation of a neural network, applying the weighting parameter to an output vector of the candidate operation, performing a regularization of the weighting parameter output vector combination, compressing the neural network model according to the results of the regularization, and providing the neural network model after compression.

First claim

What is claimed is:

1. A computer implemented method for domain specific model compression, the method comprising:
providing, by one or more computer processors, a pre-trained neural network model;
providing, by the one or more computer processors, an additional weighting parameter for a candidate operation of the pre-trained neural network model;
applying, by the one or more computer processors, the weighting parameter to an output vector of the candidate operation;
performing, by the one or more computer processors, a regularization of the application of the weighting parameter to the output vector, wherein the regularization comprises:
adding a penalty term to a loss function, the penalty term associated with the weighting parameter, wherein the penalty term includes one of an absolute value of the additional weighting parameter and the square of the additional weighting parameter;
compressing, by the one or more computer processors, the neural network model according to a result of the regularization, by removing candidate operations having a zero-value weighting parameter; and
providing, by the one or more computer processors, the neural network model after compression.

2. The computer implemented method according to claim 1 , wherein the neural network model comprises a language processing model.

3. The computer implemented method according to claim 1 , further comprising training, by the one or more computer processors, the neural network using unlabeled domain data.

4. The computer implemented method according to claim 1 , further comprising training, by the one or more computer processors, the neural network with two objectives, wherein one objective comprises a domain classification task.

5. The computer implemented method according to claim 1 , further comprising training, by the one or more computer processors, the neural network using labeled domain data.

6. The computer implemented method according to claim 1 , further comprising reducing, by the one or more computer processors, a neural network model attention head weighting value to zero.

7. The computer implemented method according to claim 1 , further comprising:
training, by the one or more computer processors, the neural network with two objectives, wherein one objective comprises a domain classification task; and
training, by the one or more computer processors, the neural network using labeled data.

8. A computer program product for domain specific model compression, the computer program product comprising one or more computer readable storage devices and stored program instructions on the one or more computer readable storage devices, the stored program instructions comprising:
program instructions to provide a pre-trained neural network model;
program instructions to provide an additional weighting parameter for a candidate operation of the neural network model;
program instructions to apply the weighting parameter to an output vector of the candidate operation;
program instructions to perform a regularization of the application of the weighting parameter to the output vector wherein the regularization comprises:
adding a penalty term to a loss function, the penalty term associated with the weighting parameter, wherein the penalty term includes one of an absolute value of the additional weighting parameter and the square of the additional weighting parameter;
program instructions to compress the neural network model according to a result of the regularization, by removing candidate operations having a zero-value weighting parameter; and
program instructions to provide the neural network model after compression.

9. The computer program product according to claim 8 , wherein the neural network model comprises a language processing model.

10. The computer program product according to claim 8 , further comprising program instructions to train the neural network using unlabeled domain data.

11. The computer program product according to claim 8 , further comprising program instructions to train the neural network with two objectives, wherein one objective comprises a domain classification task.

12. The computer program product according to claim 8 , further comprising program instructions to train the neural network using labeled domain data.

13. The computer program product according to claim 8 , further comprising program instructions to reduce a neural network model attention head weighting value to zero.

14. The computer program product according to claim 8 , further comprising program instructions to:
train the neural network with two objectives, wherein one objective comprises a domain classification task; and
train the neural network using labeled data.

15. A computer system for domain specific model compression, the computer system comprising:
one or more computer processors;
one or more computer readable storage devices; and
stored program instructions on the one or more computer readable storage devices for execution by the one or more computer processors, the stored program instructions comprising:
program instructions to provide a pre-trained neural network model;
program instructions to provide an additional weighting parameter for a candidate operation of the neural network model;
program instructions to apply the weighting parameter to an output vector of the candidate operation;
program instructions to perform a regularization of the application of the weighting parameter to the output vector wherein the regularization comprises:
adding a penalty term to a loss function, the penalty term associated with the weighting parameter, wherein the penalty term includes one of an absolute value of the additional weighting parameter and the square of the additional weighting parameter;
program instructions to compress the neural network model according to a result of the regularization, by removing candidate operations having a zero-value weighting parameter; and
program instructions to provide the neural network model after compression.

16. The computer system according to claim 15 , wherein the neural network model comprises a language processing model.

17. The computer system according to claim 15 , further comprising program instructions to train the neural network using unlabeled domain data.

18. The computer system according to claim 15 , further comprising program instructions to train the neural network with two objectives, wherein one objective comprises a domain classification task.

19. The computer system according to claim 15 , further comprising program instructions to train the neural network using labeled domain data.

20. The computer system according to claim 15 , further comprising program instructions to reduce a neural network model attention head weighting value to zero.

Assignees

• IBM

Inventors

• Wang Haoyu
• Yu Yang
• Tan Ming
• Potdar Saloni

Classifications

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title