Multivariate initialization for convolutional layers

Patent metadata

Field	Value
Publication number	US-12555198-B2
Application number	US-202318373768-A
Country	US
Kind code	B2
Filing date	Sep 27, 2023
Priority date	Sep 27, 2023
Publication date	Feb 17, 2026
Grant date	Feb 17, 2026

Abstract

A method of initializing and training a neural network having a plurality of layers includes defining a first function configured to generate a filter based on a plurality of variance values associated with respective pairs of parameters of the plurality of layers, calculating the plurality of variance values based on depths of respective layers of the plurality of layers such that the variance values increase as the depths increase, calculating a covariance matrix using the first function, the covariance matrix having a block structure and each block of the covariance matrix corresponding to a covariance between a respective parameter and other parameters of the plurality of layers, providing, as input, the covariance matrix to the neural network to initialize the neural network for training, and generating, using the neural network, an output based on the covariance matrix.

First claim

What is claimed is:

1 . A method of initializing and training a neural network having a plurality of layers, the method comprising:
defining a first function configured to generate a filter based on a plurality of variance values associated with respective pairs of parameters of the plurality of layers; calculating the plurality of variance values based on depths of respective layers of the plurality of layers such that the variance values increase as the depths increase; calculating a covariance matrix using the first function, wherein the covariance matrix has a block structure and each block of the covariance matrix corresponds to a covariance between a respective parameter and other parameters of the plurality of layers; providing, as input, the covariance matrix to the neural network to initialize the neural network for training; and generating, using the neural network, an output based on the covariance matrix.

2 . The method of claim 1 , wherein the first function corresponds to a Gaussian function.

3 . The method of claim 1 , wherein the filter is a super-Gaussian filter.

4 . The method of claim 1 , wherein the covariance corresponds to a covariance between a filter pixel and other filter pixels.

5 . The method of claim 1 , wherein calculating the plurality of variance values includes calculating the plurality of variance values further based on a velocity that indicates a rate at which the variance values increase from an initial variance as the depths increase.

6 . The method of claim 1 , wherein calculating the plurality of variance values includes calculating the plurality of variance values further based on an acceleration parameter.

7 . The method of claim 1 , wherein calculating the plurality of variance values includes calculating the plurality of variances in accordance with

σ
←

σ
0

+

v
σ

⁢
d

+

1
2

⁢

a
σ

⁢

d
2

,

wherein σ is a calculated variance value, σ 0 is an initial variance, v is a velocity at which variance grows from the initial variance as a depth d increases, and α is an acceleration parameter.

8 . The method of claim 1 , wherein the covariance matrix is used to define a multivariate probability distribution, and wherein initializing the neural network includes sampling the multivariate probability distribution to obtain initial filters for the neural network.

9 . A computing device configured to initialize and train a neural network having a plurality of layers, the computing device including a processing device configured to execute instructions stored in memory to:
define a first function configured to generate a filter based on a plurality of variance values associated with respective pairs of parameters of the plurality of layers; calculate the plurality of variance values based on depths of respective layers of the plurality of layers such that the variance values increase as the depths increase; calculate a covariance matrix using the first function, wherein the covariance matrix has a block structure and each block of the covariance matrix corresponds to a covariance between a respective parameter and other parameters of the plurality of layers; provide, as input, the covariance matrix to the neural network to initialize the neural network for training; and generate, using the neural network, an output based on the covariance matrix.

10 . The computing device of claim 9 , wherein the first function corresponds to a Gaussian function.

11 . The computing device of claim 9 , wherein the filter is a super-Gaussian filter.

12 . The computing device of claim 9 , wherein the covariance corresponds to a covariance between a filter pixel and other filter pixels.

13 . The computing device of claim 9 , wherein, to calculate the plurality of variance values, the processing device is configured to execute instructions to calculate the plurality of variance values further based on a velocity that indicates a rate at which the variance values increase from an initial variance as the depths increase.

14 . The computing device of claim 9 , wherein, to calculate the plurality of variance values, the processing device is configured to execute instructions to calculate the plurality of variance values further based on an acceleration parameter.

15 . The computing device of claim 9 , wherein, to calculate the plurality of variance values, the processing device is configured to execute instructions to calculate the plurality of variances in accordance with

σ
←

σ
0

+

v
σ

⁢
d

+

1
2

⁢

a
σ

⁢

d
2

,

wherein σ is a calculated variance value, σ 0 is an initial variance, v is a velocity at which variance grows from the initial variance as a depth d increases, and α is an acceleration parameter.

16 . The computing device of claim 9 , wherein the processing device is configured to execute instructions to use the covariance matrix to define a multivariate probability distribution, and wherein initializing the neural network includes sampling the multivariate probability distribution to obtain initial filters for the neural network.

17 . A computer-controlled machine comprising the computing device of claim 9 , wherein the computer-controlled machine further comprises an actuator configured to control an operation of the computer-controlled machine based on an output of the neural network.

18 . A system configured to train a neural network having a plurality of layers, the system comprising:
data storage that stores training data for training the neural network; memory that stores a data representation of the neural network; and a processing device configured to iteratively train the neural network using the training data to obtain a trained neural network, wherein iteratively training the neural network includes initializing the neural network by
defining a first function configured to generate a filter

Assignees

• Bosch Gmbh Robert

Inventors

• Trockman Asher
• Willmott Devin T

Classifications

• G06T5/20 Primary

  using local operators · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title