Adaptive neural network management system

Patent metadata

Field	Value
Publication number	US-10891540-B2
Application number	US-201514975420-A
Country	US
Kind code	B2
Filing date	Dec 18, 2015
Priority date	Dec 18, 2015
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

A method and computer system for managing a neural network. Data is sent into an input layer in a portion of layers of nodes in the neural network. The data moves on an encode path through the portion such that an output layer in the portion outputs encoded data. The encoded data is sent into the output layer on a decode path through the portion back to the input layer to obtain a reconstruction of the data by the input layer. A determination is made as to whether an undesired amount of error has occurred in the output layer based on the data sent into the input layer and the reconstruction of the data. A number of new nodes is added to the output layer when a determination is present that the undesired amount of the error occurred, enabling reducing the error using the number of the new nodes.

First claim

What is claimed is:

1. A method for managing a neural network, the method comprising:
sending data into an input layer in a portion of layers of nodes in the neural network in which the data moves on an encode path through the portion such that an output layer in the portion outputs encoded data;
sending the encoded data into the output layer on a decode path through the portion back to the input layer to obtain a reconstruction of the data by the input layer;
determining whether an undesired amount of error has occurred in the output layer based on the data sent into the input layer and the reconstruction of the data;
adding a number of new nodes to the output layer when a determination is present that the undesired amount of the error has occurred, enabling reducing the error using the number of the new nodes; and
training the number of new nodes by sending new data on the encode path, wherein training the new nodes does not require replay of old data.

2. The method of claim 1 , wherein determining whether the undesired amount of the error has occurred comprises:
comparing the data to the reconstruction of the data; and
determining whether a difference between the data and the reconstruction of the data is greater than a threshold for when the undesired amount of the error is present.

3. The method of claim 1 further comprising:
adding an additional layer from the layers in the neural network to the portion when the undesired amount of the error is absent, wherein the additional layer becomes the output layer, and wherein the additional layer is in a location subsequent to the output layer;
repeating sending the data into the input layer in the portion of the layers of the nodes in the neural network in which the data moves on the encode path through the portion such that the output layer in the portion outputs encoded data;
sending the encoded data into the output layer on the decode path through the portion back to the input layer to obtain the reconstruction of the data by the input layer; and
determining whether the undesired amount of the error has occurred in the output layer based on the data sent into the input layer and the reconstruction of the data.

4. The method of claim 1 , wherein the portion of the layers is a single layer and the input layer is the output layer and further comprising:
selecting another layer in the layers of the nodes as the single layer when the undesired amount of the error is absent;
sending the data into the single layer on the encode path such that the single layer outputs encoded data;
sending the encoded data into the single layer on the decode path; and
determining whether the undesired amount of the error has occurred in the single layer based on the data sent into the input layer and the reconstruction of the data by the single layer.

5. The method of claim 4 , wherein:
the steps of:
sending the data into the input layer in the portion of the layers of the nodes in the neural network in which the data moves on the encode path through the portion such that the output layer in the portion outputs encoded data;
sending the encoded data into the output layer on the decode path through the portion back to the input layer to obtain the reconstruction of the data by the input layer;
determining whether the undesired amount of the error has occurred in the output layer based on the data sent into the input layer and the reconstruction of the data; and
adding the number of the new nodes in the output layer when the determination is present that the undesired amount of the error has occurred;

are performed during a normal operation of the neural network.

6. The method of claim 1 , wherein training the number of new nodes further comprises:
creating replay data using statistics about encoding of the data by the neural network; and
sending the replay data on the encode path in addition to the new data.

7. The method of claim 1 , wherein the layers in the neural network comprise:
encode layers in the layers that encode; and
a classifier that classifies the data.

8. The method of claim 1 , wherein the neural network is a deep learning neural network.

9. The method of claim 1 , wherein weights for the new nodes are pretrained in a single-hidden-layer denoising autoencoder and adjust more quickly than weights for the nodes previously present in the network before the new nodes.

10. A computer system comprising:
a number of processors;
a number of neural network layers running on the number of processors, the neural network layers comprising:
an input layer comprising a first number of nodes;
an output layer comprising a second number of nodes;

an autoencoder running on the number of processor and configured to:
send data into a portion of the input layer on an encode path through the portion such that an output layer in the portion outputs encoded data;
send the encoded data into the output layer on a decode path through the portion back to the input layer to obtain a reconstruction of the data by the input layer; and

a neural network manager running on the number of processor and configured to:
determine whether an undesired amount of error has occurred in the output layer based on the data sent into the input layer and the reconstruction of the data;
add a number of new nodes to the output layer when a determination is present that the undesired amount of the error has occurred, enabling reducing the error using the number of the new nodes; and
send new data on the encode path, wherein the new data trains the number of the new nodes, wherein replay of old data is not required to train the new nodes.

11. The computer system of claim 10 , wherein in determining whether the undesired amount of the error has occurred, the neural network manager is configured to compare the data to the reconstruction of the data and determines whether a difference between the data and the reconstruction of the data is greater than a threshold for when the undesired amount of the error is present.

12. The computer system of claim 10 , wherein the neural network manager is configured to:
add an additional layer from the layers in the neural network to the portion of the layers in a location subsequent to the output layer in the portion when the undesired amount of the error is absent, wherein the additional layer becomes the output layer;
repeat sending the data into the input layer in the portion of the layers of the nodes in the neural network in which the data moves on the encode path through the portion such that the output layer in the portion outputs encoded data;
send the encoded data into the output layer on the decode path through the portion back to the input layer to obtain the reconstruction of the data by the input layer; and
determine whether the undesired amount of the error has occurred in the output layer based on the data sent into the input layer and the reconstruction of the data.

13. The computer system of claim 10 , wherein the portion of the layers is a single layer and the input layer is the output layer, wherein the neural network manager is configured to:
select another layer in the layers of the nodes as the single layer when the undesired amount of the error is absent;
send the data into the single layer such that the single layer outputs encoded data; sends the encoded data into the single layer on the decode path; and
determine whether the undesired amount of the error has occurred in the single layer based on the data sent into the input layer and the reconstruction of the data by the single layer.

14. The computer system of claim

Assignees

• Nat Tech & Eng Solutions Sandia Llc

Inventors

• Draelos Timothy J
• Aimone James Bradley

Classifications

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/09

  Supervised learning · CPC title