Multiobjective coevolution of deep neural network architectures

Patent metadata

Field	Value
Publication number	US-2025232174-A1
Application number	US-202519171435-A
Country	US
Kind code	A1
Filing date	Apr 7, 2025
Priority date	Nov 1, 2018
Publication date	Jul 17, 2025
Grant date	—

Abstract

An evolutionary AutoML framework called LEAF optimizes hyperparameters, network architectures and the size of the network. LEAF makes use of both evolutionary algorithms (EAs) and distributed computing frameworks. A multiobjective evolutionary algorithm is used to maximize the performance and minimize the complexity of the evolved networks simultaneously by calculating the Pareto front given a group of individuals that have been evaluated for multiple objectives.

First claim

We claim:

1 . An automated machine learning (AutoML) implementation system for evolving deep neural network (DNNs) to perform a predetermined task, comprising:
an algorithm layer for evolving deep neural network (DNN) hyperparameters and architectures, the algorithm including instructions for initializing a first population DNNs each DNN comprised of modules and a blueprint containing a combination of one or more existing species of modules, wherein for each existing species of modules in the first population and during each generation in a plurality of generations of subsequent populations:
creating a first set of empty species of modules;
for each existing species of modules, determining Pareto front of DNNs for each existing species in accordance with a fitness value and a novelty score;
removing DNNs in the Pareto front of each existing species and adding them to the first set of empty species to form one or more sets of new species;
replacing each existing species with the one more sets of new species;
truncating the one or more sets of new species by preserving a predetermined fraction of modules within each set of new species of modules, wherein the modules within each set of new species of modules are ranked in accordance with fitness value and novelty score and the predetermined fraction of modules is preserved from the modules with the highest rankings;
generating new DNNs using procreation, including at least one of crossover and mutation;
adding new DNNs to the first set of new species;

a system layer for parallel training of multiple evolved DNNs received from the algorithm layer, determination of one or more fitness values and one or more novelty scores for each of the received DNNs, and providing the one or more fitness values and one or more novelty scores back to the algorithm layer for use in additional generations of evolving DNNs; and a program layer for informing the algorithm layer and system layer of one or more desired optimization characteristics of the evolved DNNs.

2 . The AutoML implementation system of claim 1 , wherein the evolved DNNs are provided to the system layer from the algorithm layer in a JSON format.

3 . The AutoML implementation system of claim 1 , wherein the system layer for parallel training and fitness evaluation of multiple evolved DNNs includes instructions for facilitating the parallel training on a computing infrastructure with access to multiple GPU nodes.

4 . The AutoML implementation system of claim 3 , wherein the computing infrastructure is cloud-based.

5 . The AutoML implementation system of claim 3 , wherein the system layer instructions for facilitating the parallel training across the multiple GPU nodes are implemented via an API.

6 . The AutoML implementation system of claim 1 , wherein the one or more desired optimization characteristics of the evolved DNN are selected from the group consisting of: number parameters, number of floating point operations, and training time.

7 . The AutoML implementation system of claim 6 , wherein the one or more fitness values are determined in accordance with multiple objectives selected from the group consisting of performance against a benchmark and number of parameters.

8 . The AutoML implementation system of claim 1 , wherein determining the Pareto front for each existing species of DNN includes,
sorting the DNNs for each existing species in a descending order by the fitness function; creating a new Pareto front with a first DNN from each existing species;
and for each successive DNN in an existing species, if a corresponding novelty scores of a successive DNN is greater than that of a preceding DNN, adding the successive DNN to the Pareto front.

9 . The AutoML implementation system of claim 1 , further including sorting the DNNs in the Pareto front in a descending order based on the novelty score.

10 . The AutoML implementation system of claim 1 , wherein the predetermined task is a classification task.

11 . The AutoML implementation system of claim 1 , wherein the classification task is multitask image classification.

Assignees

• Cognizant Tech Solutions U S Corporation

Inventors

• Liang Jason Zhi
• Meyerson Elliot
• Miikkulainen Risto

Classifications

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title