Multiobjective coevolution of deep neural network architectures

Patent metadata

Field	Value
Publication number	US-12282845-B2
Application number	US-201916671274-A
Country	US
Kind code	B2
Filing date	Nov 1, 2019
Priority date	Nov 1, 2018
Publication date	Apr 22, 2025
Grant date	Apr 22, 2025

Abstract

An evolutionary AutoML framework called LEAF optimizes hyperparameters, network architectures and the size of the network. LEAF makes use of both evolutionary algorithms (EAs) and distributed computing frameworks. A multiobjective evolutionary algorithm is used to maximize the performance and minimize the complexity of the evolved networks simultaneously by calculating the Pareto front given a group of individuals that have been evaluated for multiple objectives.

First claim

We claim as follows:

1. A computer-implemented method for co-evolution of hyperparameters and architecture in accordance with multiple objective optimization, the method comprising:
initializing a first population of modules and a first population of blueprints containing a combination of one or more existing species of modules in the first population of modules, wherein each module is a graph including at least one node representing a deep neural network (DNN) layer and corresponding hyperparameters, and further wherein the initialization process is performed during each generation of an evolution run in a plurality of generations of subsequent populations for each existing species of modules in each of the first population of modules and the first population of blueprints:
creating a first set of modules empty of species;
for each existing species of modules, determining Pareto front of individuals for each existing species in accordance with at least a first and second objective, wherein for each individual, an individual fitness function is determined for the first objective and an individual novelty score is determined for the second objective, further wherein determining the Pareto front for each existing species of individual includes,
sorting the individuals for each existing species in a descending order by the individual fitness function;
creating a new Pareto front with a first individual from each existing species;
and for each successive individual in an existing species, if a corresponding individual novelty score of a successive individual is greater than that of a preceding individual, adding the successive individual to the Pareto front;

removing individuals in the Pareto front of each existing species and adding them to the first set of modules empty of species to form a set of new species of modules;

replacing each existing species of modules with the new species of modules;
truncating each set of the new species of modules by preserving a predetermined fraction of modules within each set of new species of modules, wherein the modules within each set of new species of modules are ranked in accordance with a module fitness function and a module novelty score and the predetermined fraction of modules is preserved from the modules with the highest rankings;
generating new individuals from the truncated sets of new species of modules using procreation, including crossover and/or mutation;
wherein the new individuals are added to a next population of modules and a next population of blueprints containing a combination of one or more existing species of modules; and
after each generation,
assembling networks using the modules and blueprints from the generated new individuals,
ranking the assembled networks in accordance with a network fitness function and a network novelty score, and
preserving a predetermined fraction of assembled networks in accordance with the ranking, wherein the predetermined fraction of assembled networks is preserved from assembled networks with the highest rankings.

2. The method of claim 1 , further including sorting the Pareto front in a descending order based on the individual novelty score.

3. The method of claim 1 , wherein the first objective is performance of the individual relative to a benchmark.

Assignees

• Cognizant Tech Solutions U S Corporation
• Cognizant Tech Solutions Us Corp

Inventors

• Liang Jason Zhi
• Meyerson Elliot
• Miikkulainen Risto

Classifications

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title