Methods and apparatus for enhancing a neural network using binary tensor and scale factor pairs

Patent metadata

Field	Value
Publication number	US-2020082264-A1
Application number	US-201816609735-A
Country	US
Kind code	A1
Filing date	May 22, 2018
Priority date	May 23, 2017
Publication date	Mar 12, 2020
Grant date	—

Abstract

Methods and apparatus are disclosed for enhancing a neural network using binary tensor and scale factor pairs. For one example, a method of optimizing a trained convolutional neural network (CNN) includes initializing an approximation residue as a trained weight tensor for the trained CNN. A plurality of binary tensors and scale factor pairs are determined. The approximation residue is updated using the binary tensors and scale factor pairs.

First claim

1 . A method of optimizing a trained convolutional neural network (CNN) comprising:
initializing an approximation residue as a trained weight tensor for the trained CNN; determining a plurality of binary tensors and scale factor pairs; and updating the approximation residue using the binary tensors and scale factor pairs.

2 . The method of claim 1 , further comprising:
iteratively determining additional binary tensors and scale factor pairs and updating the approximation residue using the additional binary tensors and scale factor pairs.

3 . The method of claim 1 , wherein iteratively determining additional binary tensors and scale factor pairs and updating the approximation residue using the additional binary tensors and scale factor pairs is repeated to find a maximum network efficiency.

4 . The method of claim 1 , further comprising:
approximating trained filters of the trained CNN by determining a basis of binary tensors and a series of scale factors.

5 . The method of claim 4 , wherein determining the basis of binary tensors and the series of scale factors includes determining a plurality of binary approximations, wherein each approximation is a combination of different binary tensors and each binary tensor is paired with a scale factor.

6 . The method of claim 1 , wherein determining the plurality of binary tensors and scale factor pairs includes:
learning heuristically the binary tensors and scale factors by selecting a first binary tensor and scale factor to be an optimum and using a previously selected optimum for each of a plurality of additional selections until all of the trained filters are approximated.

7 . The method of claim 6 , wherein each binary tensor represents a sign of a respective approximation residue and each scale factor represents a corresponding average magnitude.

8 . The method of claim 6 , wherein each binary tensor represents a sign of a respective approximation residue and each scale factor is refined using a least squares regression of all the binary tensors.

9 . The method of claim 1 , wherein the trained weight tensor comprises a floating-point weight tensor.

10 . The method of claim 1 , wherein the binary tensors and scale factor pairs comprise a binary structure.

11 . The method of claim 10 , further comprising:
using the binary structure directly in a pre-trained filter according to a CNN model to produce binary weight models via tensor expansion.

12 . The method of claim 1 , further comprising:
approximating the pre-trained filter with a linear span of a certain binary basis before initializing the approximation residue.

13 . The method of claim 1 , further comprising:
grouping identical binary tensors to pursue a maximal network efficiency.

14 . A machine-readable medium comprising instructions which when operated on by the machine cause the machine to perform a method comprising:
initializing an approximation residue as a trained weight tensor for the trained CNN; determining a plurality of binary tensors and scale factor pairs; and updating the approximation residue using the binary tensors and scale factor pairs.

15 . An apparatus comprising:
a memory to store input initial, intermediate, and final results; a neural network; and a processor to
initialize an approximation residue as a trained weight tensor for the trained CNN;
determine a plurality of binary tensors and scale factor pairs; and
update the approximation residue using the binary tensors and scale factor pairs.

16 - 29 . (canceled)

30 . The machine-readable medium of claim 14 , wherein the method further comprises:
iteratively determining additional binary tensors and scale factor pairs and updating the approximation residue using the additional binary tensors and scale factor pairs.

31 . The machine-readable medium of claim 30 , wherein iteratively determining additional binary tensors and scale factor pairs and updating the approximation residue using the additional binary tensors and scale factor pairs is repeated to find a maximum network efficiency.

32 . The apparatus of claim 15 , wherein the processor is further operable to iteratively determine additional binary tensors and scale factor pairs and update the approximation residue using the additional binary tensors and scale factor pairs.

33 . The apparatus of claim 32 , wherein the processor iteratively determines additional binary tensors and scale factor pairs and updates the approximation residue using the additional binary tensors and scale factor pairs is repeated to find a maximum network efficiency.

Assignees

• Intel Corp

Inventors

• Guo Yiwen
• Yao Anbang
• Zhao Hao
• Lu Ming
• Chen Yurong

Classifications

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title