Creating and training a second nodal network to perform a subtask of a primary nodal network

Patent metadata

Field	Value
Publication number	US-10929757-B2
Application number	US-202016923630-A
Country	US
Kind code	B2
Filing date	Jul 8, 2020
Priority date	Jan 30, 2018
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.

First claim

What is claimed is:

1. A method comprising:
training, by a computer system, a primary nodal network to perform a classification task;
after training the primary nodal network to perform the classification task, identifying, by the computer system, a sub-classification task of the classification task of the primary nodal network that will improve performance of the classification task by the primary nodal network;
training, by the computer system, a new nodal network to perform the sub-classification task; and
after training the new nodal network, merging, by the computer system, the new nodal network and the primary nodal network to create a merged network for performing the classification task.

2. The method of claim 1 , wherein merging the new and primary nodal networks comprises:
adding, by the computer system, a first set of one or more direct connections between the new and primary nodal networks; and
after adding the first set of one or more direct connections, training, by the computer system, the merged network with the first set of one or more direct connections.

3. The method of claim 2 , wherein training the merged network comprises:
initializing, by the computer system, a connection weight for each direct connection of the first set to zero; and
iteratively training, by the computer system, the merged network until the connection weight for each of the direct connections in the first set is non-zero.

4. The method of claim 3 , wherein adding one or more direct connections between the new and primary nodal networks comprises:
evaluating, by the computer system, for each of a plurality of possible direct connections between the new and primary nodal networks, a value of adding the possible direct connections; and
adding the one or more connections, by the computer system, based on the evaluation.

5. The method of claim 4 , wherein:
each of the plurality of possible direction connections is between two nodes; and
evaluating the values of adding the possible direct connections comprises computing, by the computer system, a magnitude of a sum of products of activation values for a first of the two nodes of the possible direct connection and partial derivatives of an error loss function with respect to a second of the two nodes of the direct connection, summed over a set of training data examples.

6. The method of claim 5 , wherein each direction connection in the first set is from a node in the new network to a node in the primary nodal network.

7. The method of claim 6 , further comprising, after training the merged network with the first set of one or more direct connections:
adding, by the computer system, a second set of one or more direct connections between the new and primary nodal networks, wherein the second set of one or more direction connections comprises at least one direct connection from a node of the primary nodal network to a node of the new nodal network; and
after adding the second set of one or more direct connections, training, by the computer system, the merged network with the first and second sets of one or more direct connections.

8. The method of claim 1 , wherein the sub-classification task corrects errors made by the primary nodal network on data examples.

9. The method of claim 8 , wherein training the new nodal network to perform the sub-classification task comprises training, by the computer system, the new nodal network to match a performance of another machine learning system that correctly classifies the data examples.

10. The method of claim 9 , wherein the new nodal network comprises a self-organizing partially ordered nodal network.

11. The method of claim 1 , wherein the sub-classification task provides information that causes the primary nodal network to correct an error.

12. A computer system comprising:
one or more processor cores; and
a memory in communication with the one or more processor cores, wherein the memory stores software that, when executed by the one or more processor cores, cause the one or more processor cores to:
train a primary nodal network to perform a classification task;
after training the primary nodal network to perform the classification task, identify a sub-classification task of the classification task of the primary nodal network that will improve performance of the classification task by the primary nodal network;
train a new nodal network to perform the sub-classification task; and
after training the new nodal network, merge the new nodal network and the primary nodal network to create a merged network for performing the classification task.

13. The computer system of claim 12 , wherein the memory stores further software that when executed by the one or more processors, cause the one or more processors to merge the new and primary nodal networks by:
adding a first set of one or more direct connections between the new and primary nodal networks; and
after adding the first set of one or more direct connections, training the merged network with the first set of one or more direct connections.

14. The computer system of claim 13 , wherein the memory stores further software that when executed by the one or more processors, cause the one or more processors to train the merged network by:
initializing a connection weight for each direct connection of the first set to zero; and
iteratively training the merged network until the connection weight for each of the direct connections in the first set is non-zero.

15. The computer system of claim 14 , wherein the memory stores further software that when executed by the one or more processors, cause the one or more processors to add one or more direct connections between the new and primary nodal networks by:
evaluating, for each of a plurality of possible direct connections between the new and primary nodal networks, a value of adding the possible direct connections; and
adding the one or more connections based on the evaluation.

16. The computer system of claim 15 , wherein:
each of the plurality of possible direction connections is between two nodes; and
the memory stores further software that when executed by the one or more processors, cause the one or more processors to evaluate the values of adding the possible direct connections by computing a magnitude of a sum of products of activation values for a first of the two nodes of the possible direct connection and partial derivatives of an error loss function with respect to a second of the two nodes of the direct connection, summed over a set of training data examples.

17. The computer system of claim 16 , wherein each direction connection in the first set is from a node in the new network to a node in the primary nodal network.

18. The computer system of claim 17 , wherein the memory stores further software that when executed by the one or more processors, cause the one or more processors to, after training the merged network with the first set of one or more direct connections:
add a second set of one or more direct connections between the new and primary nodal networks, wherein the second set of one or more direction connections comprises at least one direct connection from a node of the primary nodal network to a node of the new nodal network; and
after adding the second set of one or more direct connections, train the merged network with the first and second sets of one or more direct connections.

19. The computer system of claim 12 , wherein the sub-classification task corrects errors made by the primary nodal network on

Assignees

• D5Ai Llc

Inventors

• Baker James K
• Baker Bradley J

Classifications

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06N3/048

  Activation functions · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title