System and method for torque-based structured pruning for deep neural networks

Patent metadata

Field	Value
Publication number	US-2023153625-A1
Application number	US-202218052297-A
Country	US
Kind code	A1
Filing date	Nov 3, 2022
Priority date	Nov 17, 2021
Publication date	May 18, 2023
Grant date	—

Abstract

A method includes accessing a machine learning model, the machine learning model trained using a torque-based constraint. The method also includes receiving an input from an input source and providing the input to the machine learning model. The method also includes receiving an output from the machine learning model. The method also includes instructing at least one action based on the output from the machine learning model. Training the machine learning model includes applying a torque-based constraint on one or more filters of the machine learning model, adjusting, based on applying the torque-based constraint, a first set of one or more filters of the machine learning model to have a higher concentration of weights than a second set of one or more filters of the machine learning model, and pruning at least one channel of the machine learning model based on an average weight for the at least one channel.

First claim

What is claimed is:

1 . A method comprising:
accessing, using at least one processor of an electronic device, a machine learning model, the machine learning model trained using a torque-based constraint; receiving, using the at least one processor, an input from an input source; providing, using the at least one processor, the input to the machine learning model; receiving, using the at least one processor, an output from the machine learning model; and instructing, using the at least one processor, at least one action based on the output from the machine learning model.

2 . The method of claim 1 , wherein the torque-based constraint is used during training of the machine learning model to adjust a concentration of weights among a plurality of filters.

3 . The method of claim 2 , wherein the adjusting causes a weight sparsity to increase based on a filter distance from a pivot point channel.

4 . The method of claim 1 , wherein the machine learning model includes channels remaining after pruning of the machine learning model using the torque-based constraint.

5 . The method of claim 4 , wherein the pruning of the machine learning model results in a removal of at least one channel based on an average weight for the at least one channel.

6 . The method of claim 1 , wherein the input is derived from image data and the output is an image classification.

7 . An apparatus comprising:
at least one processing device configured to:
access a machine learning model, the machine learning model trained using a torque-based constraint;
receive an input from an input source;
provide the input to the machine learning model;
receive an output from the machine learning model; and
instruct at least one action based on the output from the machine learning model.

8 . The apparatus of claim 7 , wherein the torque-based constraint is used during training of the machine learning model to adjust a concentration of weights among a plurality of filters.

9 . The apparatus of claim 8 , wherein the adjusting causes a weight sparsity to increase based on a filter distance from a pivot point channel.

10 . The apparatus of claim 7 , wherein the machine learning model includes channels remaining after pruning of the machine learning model using the torque-based constraint.

11 . The apparatus of claim 10 , wherein the pruning of the machine learning model results in a removal of at least one channel based on an average weight for the at least one channel.

12 . The apparatus of claim 7 , wherein the input is derived from image data and the output is an image classification.

13 . A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:
access a machine learning model, the machine learning model trained using a torque-based constraint; receive an input from an input source; provide the input to the machine learning model; receive an output from the machine learning model; and instruct at least one action based on the output from the machine learning model.

14 . The non-transitory computer readable medium of claim 13 , wherein the torque-based constraint is used during training of the machine learning model to adjust a concentration of weights among a plurality of filters.

15 . The non-transitory computer readable medium of claim 14 , wherein the adjusting causes a weight sparsity to increase based on a filter distance from a pivot point channel.

16 . The non-transitory computer readable medium of claim 13 , wherein the machine learning model includes channels remaining after pruning of the machine learning model using the torque-based constraint.

17 . The non-transitory computer readable medium of claim 16 , wherein the pruning of the machine learning model results in a removal of at least one channel based on an average weight for the at least one channel.

18 . The non-transitory computer readable medium of claim 13 , wherein the input is derived from image data and the output is an image classification.

19 . A method comprising:
training, using at least one processor of an electronic device, a machine learning model, wherein the training includes:
applying a torque-based constraint on one or more filters of the machine learning model;
adjusting, based on applying the torque-based constraint, a first set of one or more filters of the machine learning model to have a higher concentration of weights than a second set of one or more filters of the machine learning model; and
pruning at least one channel of the machine learning model based on an average weight for the at least one channel.

20 . The method of claim 19 , wherein the adjusting causes a weight sparsity to increase based on a filter distance from a pivot point channel.

21 . The method of claim 20 , wherein the pivot point channel is an output channel of the machine learning model.

22 . The method of claim 20 , wherein the pivot point channel is an input channel of the machine learning model.

23 . The method of claim 19 , wherein the torque-based constraint is applied during a gradient update, wherein the gradient update is based on L1 regularization or based on L2 regularization.

24 . The method of claim 19 , wherein the torque-based constraint is applied as an additional term, based on L1 regularization or L2 regularization, in a loss function.

25 . The method of claim 19 , further comprising fine-tuning remaining parameters of the machine learning model.

26 . The method of claim 25 , further comprising further training the pruned machine learning model by applying an additional torque-based constraint to one or more filters of the pruned machine learning model.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Bau Tien C
• Gupta Arshita
• Garud Hrishikesh Deepak

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N5/01

  Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound · CPC title