Ensembling of neural network models

Patent metadata

Field	Value
Publication number	US-10635978-B2
Application number	US-201715794980-A
Country	US
Kind code	B2
Filing date	Oct 26, 2017
Priority date	Oct 26, 2017
Publication date	Apr 28, 2020
Grant date	Apr 28, 2020

Abstract

A method includes determining, by a processor of a computing device, a subset of models included in a plurality of models generated based on a genetic algorithm and corresponds to a first epoch of the genetic algorithm. Each of the plurality of models includes data representative of a neural network. The method includes aggregating the subset of models to generate an ensembler. The ensembler, when executed on an input, provides at least a portion of the input to each model of the subset of models to generate a plurality of intermediate outputs. An ensembler output of the ensembler is based on the plurality of intermediate outputs. The method further includes executing the ensembler on input data to determine the ensembler output.

First claim

What is claimed is:

1. A computer system comprising:
a memory configured to store an input data set and a plurality of data structures, each of the plurality of data structures including data representative of a neural network; and
a processor configured to execute a recursive search, wherein executing the recursive search comprises, during a first iteration:
determining a fitness value for each of the plurality of data structures based on at least a subset of the input data set;
determining a subset of data structures from the plurality of data structures based on the fitness values of the subset of data structures;
aggregating the subset of data structures to generate an ensembler, wherein the ensembler, when executed on an input, provides at least a portion of the input to each data structure of the subset of data structures to generate a plurality of intermediate outputs, and wherein an ensembler output of the ensembler is based on the plurality of intermediate outputs;
executing the ensembler on the input data set to determine the ensembler output;
providing the ensembler and one or more data structures of the subset of data structures as input models to a second iteration of the recursive search, the second iteration subsequent to the first iteration; and

wherein executing the recursive search, during the second iteration, comprises performing a genetic operation on the ensembler to generate a second ensembler distinct from the ensembler, wherein the genetic operation includes a crossover operation on the ensembler and a first data structure of the plurality of data structures, the first data structure representative of a first neural network; and
wherein executing the recursive search comprises outputting a final ensembler based at least in part on the second ensembler.

2. The computer system of claim 1 , wherein executing the recursive search, during the first iteration, comprises performing a genetic operation on a second data structure of the plurality of data structures.

3. The computer system of claim 1 , wherein the crossover operation includes generating a partial bit stream corresponding to a portion of a first bit stream representing the ensembler, and wherein the crossover operation includes performing crossover on the partial bit stream and a second bit stream representing the first data structure.

4. The computer system of claim 1 , wherein the ensembler output includes an output of a voting operation performed on the plurality of intermediate outputs.

5. The computer system of claim 1 , wherein each of the plurality of intermediate outputs is associated with a corresponding weight, wherein the ensembler includes a weight function generated based on the corresponding weights, and wherein the genetic operation comprises a mutation operation on the weight function.

6. The computer system of claim 5 , wherein the weight associated with a particular intermediate output is dynamically determined based on species scores of each species associated with the particular intermediate output.

7. A method comprising:
determining, by a processor of a computing device, a subset of models, the subset of models included in a plurality of models generated based on a genetic algorithm and corresponding to a first epoch of the genetic algorithm, wherein each of the plurality of models includes data representative of a neural network;
aggregating the subset of models to generate an ensembler, wherein the ensembler, when executed on an input, provides at least a portion of the input to each model of the subset of models to generate a plurality of intermediate outputs, and wherein an ensembler output of the ensembler is based on the plurality of intermediate outputs;
executing the ensembler on input data to determine the ensembler output;
providing the ensembler and one or more models of the subset of models as input models into a second epoch of the genetic algorithm, wherein the second epoch is subsequent to the first epoch;
performing a genetic operation on the ensembler during the second epoch to generate a second ensembler distinct from the ensembler, wherein the genetic operation includes a crossover operation on the ensembler and a first model of the plurality of models, the first model representative of a first neural network; and
outputting a final ensembler as an output of the genetic algorithm, wherein the final ensembler is based at least in part on the second ensembler.

8. The method of claim 7 , wherein the ensembler includes model data indicative of the subset of models and ensemble data indicative of one or more operations performed on the plurality of intermediate outputs to generate the ensembler output.

9. The method of claim 7 , wherein the first epoch and the second epoch are separated by at least one epoch.

10. The method of claim 7 , wherein the subset of models comprises elite members of elite species.

11. The method of claim 10 , wherein the plurality of models corresponds to an input population of the first epoch, and further comprising:
grouping the models of the plurality of models into species based on genetic distance between the models;
determining a species score of each of the species, wherein the species score is based on a species fitness value, a species diversity value, or both;
determining the elite species based on their respective species scores;
determining a score of each member of the elite species, wherein the score is based on a fitness value of the member, a diversity value of the member, or both; and
identifying the elite members of the elite species based on their respective scores.

12. The method of claim 7 , wherein the subset of models comprises a plurality of overall elites of the plurality of models.

13. The method of claim 12 , wherein the plurality of models corresponds to an input population of the first epoch, and further comprising:
determining a fitness value of each model of the plurality of models; and
identifying the overall elites based on their respective fitness values.

14. The method of claim 7 , further comprising:
performing at least one second genetic operation on the second ensembler during a third epoch of the genetic algorithm to generate the final ensembler, the third epoch subsequent to the second epoch.

15. The method of claim 14 , further comprising executing the second ensembler on the input data to generate a second ensembler output.

16. The method of claim 7 , wherein the genetic operation includes mutation.

17. The method of claim 7 , wherein the genetic operation is performed on model data of the ensembler, ensemble function data of the ensembler, or both, the model data indicating the subset of models, the ensemble function data indicating an ensemble function used to generate the ensembler output.

18. The method of claim 7 , wherein the final ensembler corresponds to an aggregation of a second subset of data structures.

19. A computer-readable storage device storing instructions that, when executed, cause a computer to perform operations comprising:
determining a subset of models, the subset of models included in a plurality of models generated based on a genetic algorithm and corresponding to a first epoch of the genetic algorithm, wherein each of the plurality of models includes data representative of a neural network;
aggregating the subset of models to generate an ensembler, wherein the ensembler, when executed on an input, provides at least a portion of the input to each model of the subset of models to generate a plurality of i

Assignees

• Sparkcognition Inc

Inventors

• Andoni Sari
• Moore Keith D
• Bonab Elmira M
• Choi Junhwan
• Mcdonnell Tyler S

Classifications

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/086 Primary

  using evolutionary algorithms, e.g. genetic algorithms or genetic programming · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/048

  Activation functions · CPC title