Information processing apparatus, information processing method, and program

Patent metadata

Field	Value
Publication number	US-11720786-B2
Application number	US-201715713470-A
Country	US
Kind code	B2
Filing date	Sep 22, 2017
Priority date	Sep 27, 2016
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

According to the present disclosure, a weight parameter of a neural network is divided into a plurality of portions having a certain size and approximation is individually performed on the portions using a weighted sum of the codebook vectors.

First claim

What is claimed is:

1. An information processing apparatus comprising:
one or more processors and one or more memories,
wherein the one or more processors performs, by executing programs stored in the one or more memories:
determining a plurality of three-dimensional blocks in which a four-dimensional weight parameter between a L layer and a layer next to the L layer of a neural network is divided, wherein the plurality of three-dimensional blocks are blocks in which a feature channel of the four-dimensional weight parameter having a number of channels based on a number of feature channels in the L layer and a number of feature channels in the next layer of the L layer of the neural network is divided by an integer value; and
encoding the four-dimensional weight parameter by approximating the plurality of three-dimensional blocks respectively by a linear combination of (a) codebook coefficients, and (b) two or more different three-dimensional codebook vectors, selected from a set of three-dimensional codebook vectors having a same channel size as the three-dimensional block, wherein the two or more different three-dimensional codebook vectors are part of a set of three-dimensional codebook vectors,
wherein three-dimensional codebook vectors are a predetermined number of three-dimensional codebook vectors selected with priority given to codebook vectors having larger absolute value.

2. The information processing apparatus according to claim 1 , wherein the one or more processors divides the four-dimensional weight parameter into a plurality of groups after aligning the four-dimensional weight parameter by a predetermined method.

3. The information processing apparatus according to claim 1 , wherein the four-dimensional weight parameter has elements of a binary value or a ternary value.

4. An information processing apparatus comprising:
one or more processors and one or more memories,
wherein the one or more processors performs, by executing programs stored in the one or more memories:
determining a plurality of three-dimensional blocks in which a four-dimensional weight parameter between a L layer and a layer next to the L layer of a neural network is divided, wherein the plurality of three-dimensional blocks are blocks in which a feature channel of the four-dimensional weight parameter having a number of channels based on a number of feature channels in the L layer and a number of feature channels in the next layer of the L layer of the neural network is divided by an integer value;
encoding the four-dimensional weight parameter by approximating the plurality of three-dimensional blocks respectively by a linear combination of (a) codebook coefficients, and (b) two or more different three-dimensional codebook vectors, selected from a set of three-dimensional codebook vectors having a same channel size as the three-dimensional block, wherein the two or more different three-dimensional codebook vectors are part of a set of three-dimensional codebook vectors; and
reconstructing the four-dimensional weight parameter by a linear sum of a codebook coefficient determined by the one or more processors and a corresponding codebook vector that corresponds to the codebook coefficient,
wherein a weight coefficient is determined by optimizing a loss function including a loss term of approximation accuracy of the four-dimensional weight parameter of the neural network and a loss term as a sparse term of the weight coefficient.

5. The information processing apparatus according to claim 4 ,
wherein the one or more processors reads and uses different codebook sets depending on a layer of the neural network which is a reconstruction target of the four-dimensional weight parameter.

6. The information processing apparatus according to claim 4 , wherein at least one of the weight coefficient and the codebook vector has a binary value or a ternary value as an element.

7. The information processing apparatus according to claim 4 , wherein the one or more processors further function as allowing a user to instruct a constraint condition on a learning parameter.

8. The information processing apparatus according to claim 4 , wherein the neural network is a convolutional neural network.

9. An information processing method comprising:
determining a plurality of three-dimensional blocks in which a four-dimensional weight parameter between a L layer and a layer next to the L layer of a neural network is divided, wherein the plurality of three-dimensional blocks are blocks in which a feature channel of the four-dimensional weight parameter having a number of channels based on a number of feature channels in the L layer and a number of feature channels in the next layer of the L layer of the neural network is divided by an integer value; and
encoding the four-dimensional weight parameter by approximating the plurality of three-dimensional blocks respectively by a linear combination of (a) codebook coefficients, and (b) two or more different three-dimensional codebook vectors, selected from a set of three-dimensional codebook vectors having a same channel size as the three-dimensional block, wherein the two or more different three-dimensional codebook vectors are part of a set of three-dimensional codebook vectors,
wherein three-dimensional codebook vectors are a predetermined number of three-dimensional codebook vectors selected with priority given to codebook vectors having larger absolute value.

10. A computer-readable storage medium storing a program which causes a computer to execute an information processing method, the method comprising:
determining a plurality of three-dimensional blocks in which a four-dimensional weight parameter between a L layer and a layer next to the L layer of a neural network is divided, wherein the plurality of three-dimensional blocks are blocks in which a feature channel of the four-dimensional weight parameter having a number of channels based on a number of feature channels in the L layer and a number of feature channels in the next layer of the L layer of the neural network is divided by an integer value; and
encoding the four-dimensional weight parameter by approximating the plurality of three-dimensional blocks respectively by a linear combination of (a) codebook coefficients, and (b) two or more different three-dimensional codebook vectors, selected from a set of three-dimensional codebook vectors having a same channel size as the three-dimensional block, wherein the two or more different three-dimensional codebook vectors are part of a set of three-dimensional codebook vectors,
wherein three-dimensional codebook vectors are a predetermined number of three-dimensional codebook vectors selected with priority given to codebook vectors having larger absolute value.

11. The information processing apparatus according to claim 7 , wherein the one or more processors performs learning such that the instructed constraint condition is satisfied and then encodes the four-dimensional weight parameter based on a result of the learning.

12. The information processing apparatus according to claim 11 ,
wherein the one or more processors receives, from the user, an instruction of the constraint condition regarding a memory, and
wherein the one or more processors encodes the four-dimensional weight parameter such that the four-dimensional weight parameter after compression coding becomes able to be stored into the memory.

13. The information processing apparatus according to claim 1 , wherein the one or more processors encodes the four-dimensional weight parameter using a codebook that differs depending on a pixel size of a convolution calculation in a con

Assignees

• Canon Kk

Inventors

• Tate Shunta
• Matsugu Masakazu
• Komori Yasuhiro
• Saruta Takayuki

Classifications

• G06N3/0495 Primary

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/0454

  Physics · mapped topic