Automated neural network generation using fitness estimation

Patent metadata

Field	Value
Publication number	US-10685286-B1
Application number	US-201916526360-A
Country	US
Kind code	B1
Filing date	Jul 30, 2019
Priority date	Jul 30, 2019
Publication date	Jun 16, 2020
Grant date	Jun 16, 2020

Abstract

A method of generating a neural network includes iteratively performing operations including generating, for each neural network of a population, a matrix representation. The matrix representation of a particular neural network includes rows of values, where each row corresponds to a set of layers of the particular neural network and each value specifies a hyperparameter of the set of layers. The operations also include providing the matrix representations as input to a relative fitness estimator that is trained to generate estimated fitness data for neural networks of the population. The estimated fitness data are based on expected fitness of neural networks predicted by the relative fitness estimator. The operations further include generating, based on the estimated fitness data, a subsequent population of neural networks. The method also includes, when a termination condition is satisfied, outputting data identifying a neural network as a candidate neural network.

First claim

What is claimed is:

1. A method of generating a neural network based on a data set, the method comprising:
iteratively, until a termination condition is satisfied, performing operations comprising:
generating, for each neural network of a population of neural networks, a matrix representation, wherein the matrix representation of a particular neural network includes two or more rows of values, each row corresponding to a set of layers of the particular neural network, and each value specifying a hyperparameter of the set of layers, and wherein a first matrix representation corresponding to a first neural network includes a first number of rows and a second matrix representation corresponding to a second neural network includes a second number of rows, the second number of rows different from the first number of rows;
providing each of the matrix representations as input, in a row-by-row sequential manner, to a recurrent network that is trained, based on the data set, as a relative fitness estimator to generate estimated fitness data for each of the neural networks, the estimated fitness data indicating predicted fitness of the neural networks of the population for processing the data set; and
generating, based on the population and based on the estimated fitness data, a subsequent population of neural networks; and

based on a determination that the termination condition is satisfied, outputting data identifying one or more neural networks of a final population of neural networks as a candidate neural network based on the data set.

2. The method of claim 1 , further comprising, before iteratively performing the operations:
obtaining an initial population of neural networks;
calculating, for each neural network of the initial population, a fitness value, each fitness value based on an error value generated based on output of a respective neural network of the initial population, the error value indicating a difference between one or more values of the particular data set and the output of the respective neural network;
generating training data indicating the fitness values for the neural network of the initial population and, for each fitness value, a matrix representation of a neural network associated with the fitness value; and
training a machine learning model using the training data to generate the relative fitness estimator.

3. The method of claim 2 , wherein obtaining the initial population of neural networks comprises:
determining one or more characteristics of the data set;
based on the one or more characteristics, selecting one or more parameters of an automated model generation process; and
executing the automated model generation process to generate the initial population of neural networks using a weighted randomization process, and wherein the one or more parameters weight the weighted randomization process to adjust a probability of generation of neural networks having particular features.

4. The method of claim 3 , further comprising comparing the one or more characteristics to a set of rules that maps the one or more characteristics to grammars and wherein the grammars indicate corresponding parameters.

5. The method of claim 4 , wherein the grammars further indicate the hyperparameters represented by the values of each row of the matrix representations.

6. The method of claim 1 , wherein the operations further comprise:
for one or more sample neural networks from the population, calculating a fitness value based on an error value, the error value based on output of the one or more sample neural, networks and the error value indicating a difference between one or more values of the data set and the output of the one or more sample neural networks;
generating updated training data indicating the fitness values and, for each fitness value, a matrix representation of a neural network associated with the fitness value; and
updating the relative fitness estimator using the updated training data.

7. The method of claim 1 , wherein the hyperparameters include one or more of an interconnection scheme associated with the set of layers, an activation function associated with at least one node of the set of layers, or a number of nodes of the set of layers.

8. The method of claim 1 , wherein the recurrent network includes a long short-term memory layer.

9. The method of claim 1 , wherein the two or more rows of the matrix representation of the particular neural network correspond to hidden layers of the particular neural network.

10. The method of claim 1 , wherein the subsequent population of neural networks is further based on calculated fitness values for a subset of the population of neural networks, and wherein the subset includes at least one neural network of the population of neural networks and includes fewer than all of the neural networks of the population of neural networks.

11. The method of claim 1 , wherein generating the subsequent population comprises performing evolutionary operations using one or more members of the population to generate one or more members of the subsequent population, the one or more members of the population selected based on the estimated fitness data.

12. The method of claim 1 , wherein generating the subsequent population comprises training one or more members of the population to generate one or more members of the subsequent population, the one or more members of the population selected based on the estimated fitness data.

13. A computing device comprising:
a processor; and
a memory storing instructions executable by the processor to cause the processor to iteratively, until a termination condition is satisfied, perform operations comprising:
generating, for each neural network of a population of neural networks, a matrix representation, wherein the matrix representation of a particular neural network includes two or more rows of values, each row corresponding to a set of layers of the particular neural network, and each value specifying a hyperparameter of the set of layers, and wherein a first matrix representation corresponding to a first neural network includes a first number of rows and a second matrix representation corresponding to a second neural network includes a second number of rows, the second number of rows different from the first number of rows;
providing each of the matrix representations as input, in a row-by-row sequential manner, to a recurrent network that is trained, based on a data set, as a relative fitness estimator to generate estimated fitness data for each of the neural networks the estimated fitness data indicating predicted fitness of the neural networks of the population for processing the data set; and
generating, based on the population and based on the estimated fitness data, a subsequent population of neural networks; and
the instructions further executable by the processor to cause the processor to, based on a determination that the termination condition is satisfied, output data identifying one or more neural networks of a final population of neural networks as a candidate neural network.

14. The computing device of claim 13 , wherein the instructions are further executable by the processor to cause the processor to, before iteratively performing the operations:
obtain an initial population of neural networks;
calculate, for each neural network of the initial population, a fitness value, each fitness value based on an error value, the error value generated based on output of a respective neural network of the initial population and the error value indicating a difference between one or more values of the particular data set and the

Assignees

• Sparkcognition Inc

Inventors

• Mcdonnell Tyler S
• Greenwood Bryson

Classifications

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title