Evolutionary architectures for evolution of deep neural networks

Patent metadata

Field	Value
Publication number	US-11003994-B2
Application number	US-201816212830-A
Country	US
Kind code	B2
Filing date	Dec 7, 2018
Priority date	Dec 13, 2017
Publication date	May 11, 2021
Grant date	May 11, 2021

Abstract

A system and method for evolving a deep neural network structure that solves a provided problem includes: a memory storing a candidate supermodule genome database having a pool of candidate supermodules having values for hyperparameters for identifying a plurality of neural network modules in the candidate supermodule and further storing fixed multitask neural networks; a training module that assembles and trains N enhanced fixed multitask neural networks and trains each enhanced fixed multitask neural network using training data; an evaluation module that evaluates a performance of each enhanced fixed multitask neural network using validation data; a competition module that discards supermodules in accordance with assigned fitness values and saves others in an elitist pool; an evolution module that evolves the supermodules in the elitist pool; and a solution harvesting module providing for deployment of a selected one of the enhanced fixed multitask neural networks, instantiated with supermodules selected from the elitist pool.

First claim

The invention claimed is:

1. A computer-implemented system for evolving a deep neural network structure that solves a provided problem, the system comprising:
a memory storing a candidate supermodule genome database having a pool of candidate supermodules, each of the candidate supermodules identifying respective values for a plurality of supermodule hyperparameters of the supermodule, the supermodule hyperparameters including supermodule global topology hyperparameters identifying a plurality of neural network modules in the candidate supermodule and module interconnects among the neural network modules in the candidate supermodule, each candidate supermodule having associated therewith storage for an indication of a respective supermodule fitness value;
the memory further storing soft order neural networks;
a training module that assembles and trains N enhanced soft order neural networks by:
selecting a population of K supermodules from the pool of candidate supermodules, the population of K supermodules including M species of supermodules;
initializing a population of N soft order neural networks;
randomly selecting supermodules from each M species of supermodules of the population of K supermodules to create N sets of supermodules, the supermodules being selected such that each set of supermodules includes a supermodule from each of the M species of supermodules,
assembling each set of supermodules of the N sets of supermodules with a corresponding soft order neural network of the population of N soft order neural network to obtain N assembled enhanced soft order neural networks, and

training each enhanced soft order neural network using training data;
an evaluation module that evaluates a performance of each enhanced soft order neural network using validation data to (i) determine an enhanced soft order neural network fitness value for each enhanced soft order neural network and (ii) assigns a determined enhanced soft order neural network fitness value to corresponding neural network modules in the selected population of K supermodules;
a competition module that discards supermodules from the population of K supermodules in dependence on their assigned fitness values and stores the remaining supermodules in an elitist pool;
an evolution module that evolves the supermodules in the elitist pool; and
a solution harvesting module providing for deployment of a selected one of the enhanced soft order neural networks, instantiated with supermodules selected from the elitist pool.

2. The system of claim 1 , wherein each of the enhanced soft order neural networks has a set number of nodes and a set number of slots and each neural network module of a respective neural network supermodule is of a same topology.

3. The system of claim 1 , wherein the enhanced soft ordered neural networks have a fixed grid-like structure.

4. The system of claim 1 , wherein a flag that is associated with each slot in the enhanced soft ordered neural networks indicates whether a corresponding neural network module will share corresponding weights and accept shared weights from other neural network modules in the slot or other slots.

5. A computer-implemented system for evolving a deep neural network structure that solves a provided problem, the system comprising:
a memory storing a candidate supermodule genome database having a pool of candidate supermodules, each of the candidate supermodules identifying respective values for a plurality of supermodule hyperparameters of the supermodule, the supermodule hyperparameters including supermodule global topology hyperparameters identifying a plurality of neural network modules in the candidate supermodule and module interconnects among the neural network modules in the candidate supermodule, each candidate supermodule having associated therewith storage for an indication of a respective supermodule fitness value;
the memory further storing fixed multitask neural networks;
a training module that assembles and trains N enhanced fixed multitask neural networks by:
selecting a population of K supermodules from the pool of candidate supermodules, the population of K supermodules including M species of supermodules;
initializing a population of N fixed multitask neural networks;
randomly selecting supermodules from each M species of supermodules of the population of K supermodules to create N sets of supermodules, the supermodules being selected such that each set of supermodules includes a supermodule from each of the M species of supermodules,
assembling each set of supermodules of the N sets of supermodules with a corresponding fixed multitask neural network of the population of N fixed multitask neural network to obtain N assembled enhanced fixed multitask neural networks, and
training each enhanced fixed multitask neural network using training data;

an evaluation module that evaluates a performance of each enhanced fixed multitask neural network using validation data to (i) determine an enhanced fixed multitask neural network fitness value for each enhanced fixed multitask neural network and (ii) assigns a determined enhanced fixed multitask neural network fitness value to corresponding neural network modules in the selected population of K supermodules;
a competition module that discards supermodules from the population of K supermodules in dependence on their assigned fitness values and stores the remaining supermodules in an elitist pool;
an evolution module that evolves the supermodules in the elitist pool; and
a solution harvesting module providing for deployment of a selected one of the enhanced fixed multitask neural networks, instantiated with supermodules selected from the elitist pool.

6. A computer-implemented system for evolving a deep neural network structure that solves a provided problem, the system comprising:
a memory storing a candidate supermodule genome database having a pool of candidate supermodules, each of the candidate supermodules identifying respective values for a plurality of supermodule hyperparameters of the supermodule, the supermodule hyperparameters including supermodule global topology hyperparameters identifying a plurality of neural network modules in the candidate supermodule and module interconnects among the neural network modules in the candidate supermodule, each candidate supermodule having associated therewith storage for an indication of a respective supermodule fitness value;
the memory further storing a blueprint genome database having a pool of candidate blueprints for solving the provided problem, each of the candidate blueprints identifying respective values for a plurality of blueprint topology hyperparameters of the blueprint, the blueprint topology hyperparameters including a number of included supermodules, and interconnects among the included supermodules, each candidate blueprint having associated therewith storage for an indication of a respective blueprint fitness value;
a training module that assembles and trains N neural networks by:
selecting a population of N candidate blueprints from the pool of candidate blueprints,
randomly selecting, for each candidate blueprint of the population of N candidate blueprints and from the pool of candidate supermodules, a corresponding set of supermodules for each species of a plurality of species represented by the pool of candidate supermodules,
assembling each of the N candidate blueprints with their corresponding set of supermodules to obtain the N neural networks, wherein each node of each candidate blueprint is replaced by a supermodule of their corresponding set of supermodules and wherein, if a neural network module of a supermodule has multiple inputs from a previous node, then the inputs are soft merged together, and
training each of the N neural ne

Assignees

• Cognizant Tech Solutions U S Corporation

Inventors

• Liang Jason Zhi
• Meyerson Elliot
• Miikkulainen Risto

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title