Apparatus and method

Patent metadata

Field	Value
Publication number	US-2018253401-A1
Application number	US-201815903290-A
Country	US
Kind code	A1
Filing date	Feb 23, 2018
Priority date	Mar 2, 2017
Publication date	Sep 6, 2018
Grant date	—

Abstract

An apparatus comprising circuitry that implements an artificial neural network training algorithm that uses weight tying.

First claim

1 . An apparatus comprising circuitry that implements an artificial neural network training algorithm that uses weight tying.

2 . The apparatus of claim 1 , wherein the circuitry is configured to update the weight tying using a predefined number of iterations of a clustering algorithm.

3 . The apparatus of claim 1 , wherein the circuitry is configured to compute a weight-tied weight matrix based on an index matrix and based on a value vector.

4 . The apparatus of claim 2 , wherein the predefined number of iterations of the clustering algorithm used to update the weight tying is one.

5 . The apparatus of claim 2 , wherein the circuitry is configured to, in each iteration of the clustering algorithm, update a value vector according to

[

v

(
l
)

]

k

=

1

#


{

I

(
l
)

=
k

}



∑

ij
,

I

(
l
)

=
k



[

W

(
l
)

]

ij

where W (l) is a full-precision weight matrix for layer l of the neural network, and I (l) is the index matrix.

6 . The apparatus of claim 2 , wherein the circuitry is configured to update, in each iteration of the clustering algorithm, an index matrix according to
[ I (l) ] ij =arg min k=1, . . . , K (l) |[W (l) ] ij −[v (l) ] k |

7 . The apparatus of claim 3 , wherein the circuitry is configured to quantize the values of the value vector.

8 . The apparatus of claim 3 , wherein the circuitry is configured to quantize the values of the value vector after updating the weight tying.

9 . The apparatus of claim 7 , wherein the circuitry is configured to quantize the values of the value vector to the nearest power-of-two.

10 . The apparatus of claim 9 , wherein the circuitry is configured to quantize the values of the value vector according to the quantization scheme:

x
q

=

{

s
·

2

⌊
b
⌋

b
-

⌊
b
⌋

≤

log
2


1.5

s
·

2

⌈
b
⌉

b
-

⌊
b
⌋

>

log
2


1.5

where s=sign(x) and b=log 2 |x|, and where x is the value which is to be quantized and x q is the quantized value.

11 . The apparatus of claim 3 , wherein a value vector comprises more than three values.

12 . The apparatus of claim 1 , wherein the circuitry is configured to update full precision weights based on gradients.

13 . The apparatus of claim 12 , wherein the circuitry is configured to compute the gradients based on a cost function and based on the weight-tied weight matrix.

14 . The apparatus of claim 12 , wherein the circuitry is configured to compute the cost function based on a loss function and based on a forward pass function.

15 . The apparatus of claim 12 , wherein the circuitry is configured to compute the gradients based on a backward pass function.

16 . The apparatus of claim 1 , wherein the training algorithm is a stochastic gradient descent training algorithm.

17 . The apparatus of claim 1 , wherein the artificial neural network is a deep convolutional neural network.

18 . An apparatus comprising circuitry that implements an artificial neural network, the artificial neural network having been trained by a neural network training algorithm that uses weight tying.

19 . The apparatus of claim 18 , wherein in the circuitry implements the artificial neural network multiplierless.

20 . A method of training an artificial neural network, the method comprising performing an artificial neural network training algorithm that uses weight tying.

Assignees

• Sony Corp

Inventors

• Cardinaux Fabien
• Uhlich Stefan
• Kemp Thomas
• Alonso Garcia Javier
• Yoshiyama Kazuki

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/063

  using electronic means · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title