Extension of the capsule network

Patent metadata

Field	Value
Publication number	US-2019303742-A1
Application number	US-201815943445-A
Country	US
Kind code	A1
Filing date	Apr 2, 2018
Priority date	Apr 2, 2018
Publication date	Oct 3, 2019
Grant date	—

Abstract

A capsule neural network is extended to include a factorization machine and two factor matrices. The factor matrices can be low rank matrices that are substituted for a trainable matrix in conventional capsules. The factor matrices can be differentially trained using the factorization machine. Because the factor matrices have substantially fewer elements than the trainable matrix, a capsule network can be trained in less time and use less memory than required for conventional capsule networks.

First claim

What is claimed is:

1 . A method comprising:
instantiating a capsule network having a plurality of capsules arranged in one or more layers, wherein each capsule includes a trainable transformation matrix; receiving a value for a factor matrix inner dimension; determining a first factor matrix and a second factor matrix for a capsule, wherein the first factor matrix and the second factor matrix have dimensions based on dimensions of the trainable transformation matrix and the factor matrix inner dimension; receiving training data for the capsule network; comparing actual output of the capsule network with desired output associated with the training data; determining a system of equations associated with the first factor matrix and the second factor matrix based, at least in part, on differences determined by comparison of the actual output with the desired output; and supplying the system of equations to a factorization machine to determine updated values for entries in the first factor matrix and the second factor matrix.

2 . The method of claim 1 further comprising:
reconstructing the trainable transformation matrix using the first factor matrix and the second factor matrix.

3 . The method of claim 1 , wherein the value of the factor matrix inner dimension is greater than or equal to three (3) and less than or equal to six (6).

4 . The method of claim 1 , further comprising:
Increasing the value of the factor matrix inner dimension based on determining that the capsule network is not converging.

5 . The method of claim 1 , further comprising configuring the factorization machine to utilize stochastic gradient descent as a learning mode.

6 . One or more non-transitory machine-readable media comprising program code for training a capsule network, the program code to:
instantiate a capsule network having a plurality of capsules arranged in one or more layers, wherein each capsule includes a trainable transformation matrix; receive a value for a factor matrix inner dimension; determine a first factor matrix and a second factor matrix for a capsule, wherein the first factor matrix and the second factor matrix have dimensions based on dimensions of the trainable transformation matrix and the factor matrix inner dimension; receive training data for the capsule network; compare actual output of the capsule network with desired output associated with the training data; determine a system of equations associated with the first factor matrix and the second factor matrix based, at least in part, on differences determined by comparison of the actual output with the desired output; and supply the system of equations to a factorization machine to determine updated values for entries in the first factor matrix and the second factor matrix.

7 . The one or more non-transitory machine-readable media of claim 6 , wherein the program code further includes program code to:
reconstruct the trainable transformation matrix using the first factor matrix and the second factor matrix.

8 . The one or more non-transitory machine-readable media of claim 6 , wherein the value of the factor matrix inner dimension is greater than or equal to three (3) and less than or equal to six (6).

9 . The one or more non-transitory machine-readable media of claim 6 , wherein the program code further comprises program code to:
increase the value of the factor matrix inner dimension based on a determination that the capsule network is not converging.

10 . The one or more non-transitory machine-readable media of claim 6 , wherein the program code further comprises program code to configure the factorization machine to utilize stochastic gradient descent as a learning mode.

11 . An apparatus comprising:
at least one processor; and a non-transitory machine-readable medium having program code executable by the at least one processor to cause the apparatus to,
instantiate a capsule network having a plurality of capsules arranged in one or more layers, wherein each capsule includes a trainable transformation matrix;
receive a value for a factor matrix inner dimension;
determine a first factor matrix and a second factor matrix for a capsule, wherein the first factor matrix and the second factor matrix have dimensions based on dimensions of the trainable transformation matrix and the factor matrix inner dimension;
receive training data for the capsule network;
compare actual output of the capsule network with desired output associated with the training data;
determine a system of equations associated with the first factor matrix and the second factor matrix based, at least in part, on differences determined by comparison of the actual output with the desired output; and
supply the system of equations to a factorization machine to determine updated values for entries in the first factor matrix and the second factor matrix.

12 . The apparatus of claim 11 , wherein the program code further includes program code to:
reconstruct the trainable transformation matrix using the first factor matrix and the second factor matrix.

13 . The apparatus of claim 11 , wherein the value of the factor matrix inner dimension is greater than or equal to three (3) and less than or equal to six (6).

14 . The apparatus of claim 11 , wherein the program code further comprises program code to:
increase the value of the factor matrix inner dimension based on a determination that the capsule network is not converging.

15 . The apparatus of claim 11 , wherein the program code further comprises program code to configure the factorization machine to utilize stochastic gradient descent as a learning mode.

Assignees

• Ca Inc

Inventors

• Bonnell Christopher Phillip

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title