Accelerated training of a neural network via regularization

Patent metadata

Field	Value
Publication number	US-12430559-B2
Application number	US-202519098299-A
Country	US
Kind code	B2
Filing date	Apr 2, 2025
Priority date	Jan 30, 2018
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

First claim

What is claimed is:

1. A method of accelerating training a first neural network, wherein the first neural network comprises multiple nodes, wherein the multiple nodes comprise a first node, the method comprising:
iterative training, by a programmed computer system that comprises one or more processor cores, via machine learning, the first neural network over a first number of iterations, wherein the iterative training comprises:
for each training data item in a set of training data items:
in a forward propagation phase, determining an activation values for each of the multiple nodes; and
in a back-propagation phase, determining estimated partial derivatives of learned parameters and activation values for the multiple nodes with respect to, at least, an objective of the first neural network; and

updating the learned parameters associated with the multiple nodes based on the estimated partial derivatives,

wherein determining the estimated partial derivatives comprises determining estimated partial derivatives for the first node, and wherein determining the estimated partial derivatives for the first node comprises:
adding by the programmed computer system, a regularization cost term to the objective for a node state of the first node for a first training data item in the set of training data items, wherein the regularization cost term is based on a difference between the node state of the first node for the first training data item and a node state of a second node for the first training data item, wherein the second node is one of multiple nodes of a second neural network that is separate from the first neural network.

2. The method of claim 1 , wherein the node state of the first node comprises a first learned parameter for the first node and the node state for the second node comprises a second learned parameter for the second node.

3. The method of claim 2 , wherein the first and second learned parameters comprise connection weights.

4. The method of claim 1 , wherein the node state of the first node comprises a first activation value for the first node and the node state for the second node comprises a second activation value for the second node.

5. The method of claim 1 , wherein the regularization cost term is a squared difference between the node state of the first node and the node state of the second node.

6. The method of claim 5 , wherein the regularization cost term is multiplied by a hyperparameter that controls a strength of the regularization.

7. The method of claim 1 , wherein the regularization cost term is an absolute difference between the node state of the first node and the node state of the second node.

8. The method of claim 1 , wherein the regularization cost term is multiplied by a hyperparameter that controls a strength of the regularization.

9. The method of claim 1 , wherein the first neural network and the second neural network have different architectures.

10. The method of claim 1 , further comprising, prior to iterative training the first neural network, iteratively training, via machine learning, by the programmed computer system, the second neural network to convergence.

11. The method of claim 1 , wherein updating the learned parameters comprises updating the learned parameters via stochastic gradient descent.

12. The method of claim 1 , wherein updating the learned parameters comprises updating the learned parameters via gradient descent.

13. The method of claim 1 , wherein the first node is an output node of the first neural network and the second node is an output node of the second neural network.

14. The method of claim 1 , wherein the first node is a labeled feature node of the first neural network and the second node is a labeled feature node of the second neural network.

15. The method of claim 1 , further comprising, after iterative training the first neural network, merging, by the programmed computer system, the first and second neural networks.

16. The method of claim 1 , wherein the first and second neural networks are members of an ensemble.

17. A computer system for accelerated training of a first neural network, wherein the first neural network comprises multiple nodes, wherein the multiple nodes comprise a first node, the computer system comprising:
one or more processor cores; and
computer memory in communication with the one or more processor cores, wherein the computer stores instructions that when executed by the one or more processor cores cause the one or more processor cores to perform iterative training, via machine learning, of the first neural network, wherein the iterative training comprises:
for each training data item in a set of training data items:
in a forward propagation phase, determining an activation values for each of the multiple nodes; and
in a back-propagation phase, determining estimated partial derivatives of learned parameters and activation values for the multiple nodes, including the first node, with respect to, at least, an objective of the first neural network; and

updating the learned parameters associated with the multiple nodes based on the estimated partial derivatives,

wherein the computer memory stores instructions that when executed by the one or more processor cores cause the one or more processor cores to determine the estimated partial derivatives for the first node by adding a regularization cost term to the objective for a node state of the first node for a first training data item in the set of training data items, wherein the regularization cost term is based on a difference between the node state of the first node for the first training data item and a node state of a second node for the first training data item, wherein the second node is one of multiple nodes of a second neural network that is separate from the first neural network.

18. The computer system of claim 17 , wherein the node state of the first node comprises a first learned parameter for the first node and the node state for the second node comprises a second learned parameter for the second node.

19. The computer system of claim 18 , wherein the first and second learned parameters comprise connection weights.

20. The computer system of claim 17 , wherein the node state of the first node comprises a first activation value for the first node and the node state for the second node comprises a second activation value for the second node.

21. The computer system of claim 17 , wherein the regularization cost term is a squared difference between the node state of the first node and the node state of the second node.

22. The computer system of claim 21 , wherein the regularization cost term is multiplied by a hyperparameter that controls a strength of the regularization.

23. The computer system of claim 17 , wherein the regularization cost term is an absolute difference between the node state of the first node and the node state of the second node.

24. The computer system of claim 17 , wherein the regularization cost term is multiplied by a hyperparameter that controls a strength of the regularization.

25. The computer system of claim 17 , wherein the first neural network and the second neural network have different architectures.

26. The computer system of claim 17 , wherein the computer memory stores further instructions that when executed by the one or more processors cause the one or more processors to, prior to iterative training

Assignees

• D5Ai Llc

Inventors

• Baker James K
• Baker Bradley J

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title