Neural network system that executes batch normalization

Patent metadata

Field	Value
Publication number	US-2024256867-A1
Application number	US-202118560798-A
Country	US
Kind code	A1
Filing date	Sep 24, 2021
Priority date	Jun 9, 2021
Publication date	Aug 1, 2024
Grant date	—

Abstract

A neural network system includes: a first layer generating first layer outputs of pieces of training data; a second layer; and a numerical conversion layer. During training, the numerical conversion layer: receives the first layer outputs from the first layer, calculates a numerical conversion parameter corresponding to each of the pieces of training data, numerically converts each of components of each of the first layer outputs using the numerical conversion parameter to generate a numerical conversion layer output, and inputs the same to the second layer. The numerical conversion parameter corresponding to one of the pieces of training data is: calculated from the first layer outputs of the pieces of training data except the one of the pieces of training data, or calculated by weighting one of the first layer outputs of the pieces of training data including the one of the pieces of training data.

First claim

1 . A neural network system implemented by one or more computers, comprising:
a first layer that generates first layer outputs of a plurality of pieces of training data,
each of the first layer outputs having a plurality of components;
a second layer; and a numerical conversion layer disposed between the first layer and the second layer, wherein during training of the neural network system, the numerical conversion layer:
receives the first layer outputs from the first layer,
calculates a numerical conversion parameter corresponding to each of the pieces of training data,
numerically converts each of the components of each of the first layer outputs using the numerical conversion parameter to generate a numerical conversion layer output, and
inputs the numerical conversion layer output to the second layer, and
the numerical conversion parameter corresponding to one of the pieces of training data is: calculated from the first layer outputs of the pieces of training data except the one of the pieces of training data, or
calculated by weighting each of the first layer outputs of the pieces of training data including the one of the pieces of training data, and
a weight of one of the first layer outputs of the one of the pieces of training data is smaller than a weight of the other first layer outputs of the other pieces of training data.

2 . The neural network system according to claim 1 , wherein the numerical conversion parameter corresponding to the one of the pieces of training data is calculated from first layer outputs of a plurality of pieces of training data selected, from a batch including a set of the pieces of training data including the one of the pieces of training data, by a predetermined selection method to exclude the one of the pieces of training data.

3 . The neural network system according to claim 2 , wherein
the components of each of the first layer outputs are indexed by dimensions, and the numerical conversion layer calculates the numerical conversion parameter by:
with respect to the pieces of training data per the batch, for each of the dimensions, calculating a mean of the components of each of the first layer outputs of the pieces of training data selected by the selection method, as a pseudo mean of the components of each of the first layer outputs, and
calculating, for each of the dimensions, a variance of the components of each of the first layer outputs using the components of each of the first layer outputs and the pseudo mean with respect to the pieces of training data per the batch.

4 . The neural network system according to claim 3 , wherein the numerical conversion layer generates the numerical conversion layer output by, for each of the pieces of training data, numerically converting the components of each of the first layer outputs of the pieces of training data using the pseudo mean and the variance for each of the dimensions corresponding to each of the components.

5 . The neural network system according to claim 4 , wherein the numerical conversion layer generates the numerical conversion layer output by of transforming the components numerically converted based on values of a set of transformation parameters for each of the dimensions.

6 . The neural network system according to claim 5 , wherein
after the neural network system is trained, the numerical conversion layer:
receives a new first layer output for a new neural network input generated by the first layer,
generates a new numerically converted layer output by numerically converting each of components of the new first layer output using a precalculated numerical conversion parameter,
generates a new numerical conversion layer output by converting, for each of the dimensions, each of components of the new numerically converted layer output based on a set of transformation parameters for each of the dimensions, and
newly inputs the new numerical conversion layer output to the second layer.

7 . The neural network system according to claim 6 , wherein the precalculated numerical conversion parameter is calculated from the first layer outputs generated by the first layer during the training of the neural network system.

8 . The neural network system according to claim 6 , wherein the precalculated numerical conversion parameter is calculated from the new first layer output generated by the first layer after the neural network system is trained.

9 . The neural network system according to claim 7 , wherein a new neural network input processed by the neural network system after the neural network system is trained is an input of a different type from the pieces of training data used to train the neural network system.

10 . The neural network system according to claim 2 , wherein
the components of each of the first layer outputs are indexed by a feature index and a spatial location index, the numerical conversion layer calculates the numerical conversion parameter by:
with respect to the pieces of training data per the batch, for each combination of the feature index and the spatial location index, calculating a mean of the components of each of the first layer outputs of the pieces of training data selected by the selection method,
with respect to the pieces of training data per the batch, for each of the feature index, calculating an arithmetic mean of the mean with respect to the combination including the feature index,
for each combination of the feature index and the spatial location index, calculating a variance of the components of each of the first layer outputs using the components of each of the first layer outputs and the arithmetic mean with respect to the pieces of training data per the batch, and
for each of the feature index, calculating an arithmetic variance of the variance with respect to the combination including the feature index.

11 . The neural network system according to claim 10 , wherein the numerical conversion layer generates the numerical conversion layer output by, for each of the pieces of training data, numerically converting the components of each of the first layer outputs of the pieces of training data using the arithmetic mean and the arithmetic variance.

12 . The neural network system according to claim 11 , wherein the numerical conversion layer generates the numerical conversion layer output by converting the components numerically converted based on a set of transformation parameters for each of the feature index.

13 . The neural network system according to claim 12 , wherein
after the neural network system is trained, the numerical conversion layer:
receives a new first layer output for a new neural network input generated by the first layer,
generates a new numerically converted layer output by numerically converting each of components of the new first layer output using a precalculated numerical conversion parameter,
generates a new numerical conversion layer output by converting, for each of the feature index, each of components of the new numerically converted layer output based on a set of transformation parameters for each of the feature index, and
newly inputs the new numerical conversion layer output to the second layer.

14 . The neural network system according to claim 2 , wherein
the components of each of the first layer outputs are indexed by a feature index and a spatial location index, and the numerical conversion layer calculates the numerical conversion parameter by: with respect to the pieces of training data per the batch, for each of the feature index, calculating a mean of the components of each of the first layer outputs of the pi

Assignees

• Konica Minolta Inc

Inventors

• Sekii Taiki

Classifications

• G06F16/906

  Clustering; Classification · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/04 Primary

  Architecture, e.g. interconnection topology · CPC title