Automated model building search space reduction

Patent metadata

Field	Value
Publication number	US-12051002-B2
Application number	US-202016848007-A
Country	US
Kind code	B2
Filing date	Apr 14, 2020
Priority date	Nov 29, 2018
Publication date	Jul 30, 2024
Grant date	Jul 30, 2024

Abstract

A method includes receiving, by a processor, an input data set. The input data set includes a plurality of features. The method includes determining, by the processor, one or more characteristics of the input data set. The method includes, based on the one or more characteristics, adjusting, by the processor, one or more architectural parameters of an automated model generation process. The automated model generation process is configured to generate a plurality of models using a weighted randomization process. The one or more architectural parameters weight the weighted randomization process to adjust a probability of generation of models having particular architectural features. The method further includes executing, by the processor, the automated model generation process to output a mode, the model including data representative of a neural network.

First claim

What is claimed is:

1. A method of neural network generation, the method comprising:
determining, by a processor, one or more characteristics of an input data set;
based on the one or more characteristics, setting, by the processor, one or more architectural parameters to constrain topologies of hidden layers used by an automated model generation process; and
executing, by the processor, the automated model generation process to generate an initial set of models using a weighted randomization process based on the one or more architectural parameters, wherein the initial set of models comprises a model including data representative of a neural network.

2. The method of claim 1 , wherein a value of an architectural parameter of the one or more architectural parameters is used as a weighting value for the weighted randomization process.

3. The method of claim 1 , wherein probabilities of the weighted randomization process generating models having particular architectural features are determined based on weighting values associated with the particular architectural features.

4. The method of claim 1 , wherein each model of the initial set of models has a hidden layer with an architectural feature selected based on the weighted randomization process.

5. The method of claim 1 , wherein the one or more architectural parameters include a first architectural parameter associated with a probability that models of a first epoch of the weighted randomization process have a first model type.

6. The method of claim 1 , wherein a value of an architectural parameter of the one or more architectural parameters is used as a weighting value for the weighted randomization process, and wherein the weighting value is associated with a probability of generating a feedforward model, a probability of generating a recurrent model, a probability of generating a pooling-based two-dimensional convolutional model, a probability of generating a daisy-chains of causal convolutional model, or a combination thereof.

7. The method of claim 1 , further comprising, based on the one or more characteristics, adjusting, by the processor, one or more training hyperparameters of the automated model generation process, wherein the one or more training hyperparameters control one or more aspects of training of the model.

8. The method of claim 7 , wherein the training hyperparameters include one or more of a learning rate of a neural network, a momentum of a neural network, a number of epochs of the automated model generation process, or a batch size.

9. The method of claim 1 , wherein the one or more characteristics indicate a type of problem associated with the input data set, a data type associated with the input data set, or a combination thereof.

10. The method of claim 1 , wherein setting the one or more architectural parameters comprises comparing the one or more characteristics to rules that map data set characteristics to model topology parameters.

11. The method of claim 10 , further comprising, after the automated model generation process outputs the model:
determining whether a score of the model satisfies a threshold; and
responsive to determining that the score satisfies the threshold, updating one or more of the rules based on characteristics of the model.

12. The method of claim 1 , wherein executing the automated model generation process comprises:
based on a fitness function, selecting, by the processor, a subset of models from the initial set of models, the initial set of models based on a genetic algorithm and corresponding to a first epoch of the genetic algorithm;
performing, by the processor, at least one genetic operation of the genetic algorithm with respect to at least one model of the subset to generate a trainable model;
sending the trainable model to an optimization trainer to modify link weights of the trainable model; and
adding a trained model received from the optimization trainer as input to a second epoch of the genetic algorithm that is subsequent to the first epoch.

13. The method of claim 12 , wherein the fitness function is evaluated based on the input data set, and wherein the optimization trainer is configured to use a portion of the input data set to train the trainable model.

14. A computing device comprising:
one or more processors; and
a memory storing instructions executable by the one or more processors to perform operations comprising:
determining one or more characteristics of an input data set;
based on the one or more characteristics, setting one or more architectural parameters to constrain topologies of hidden layers used by an automated model generation process;
based on the one or more characteristics, adjusting one or more training hyperparameters of the automated model generation process; and
executing the automated model generation process to generate a plurality of models, wherein the plurality of models comprises a model including data representative of a neural network, and wherein the one or more training hyperparameters control one or more aspects of training of the model.

15. The computing device of claim 14 , wherein the automated model generation process generates the plurality of models using a weighted randomization process that is based on the one or more architectural parameters, and wherein probabilities of the weighted randomization process generating models having particular architectural features are determined based on weighting values associated with the particular architectural features.

16. The computing device of claim 14 , wherein executing the automated model generation process includes generating an initial set of models using a weighted randomization process based on the one or more architectural parameters.

17. The computing device of claim 16 , wherein a weighting value of the weighted randomization process is associated with a probability of generating one or more of a feedforward model, a recurrent model, a pooling-based two-dimensional convolutional model, a daisy-chains of causal convolutional model, or a combination thereof.

18. The computing device of claim 14 , wherein the one or more characteristics indicate a type of problem associated with the input data set, a data type associated with the input data set, or a combination thereof.

19. The computing device of claim 14 , wherein setting the one or more architectural parameters comprises comparing the one or more characteristics to rules that map data set characteristics to model topology parameters.

20. The computing device of claim 19 , wherein the operations further comprise updating the rules based on characteristics of the model responsive to determining that a score associated with the model satisfies a threshold.

21. A computer-readable storage device storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
determining, by the processor, one or more characteristics of an input data set;
based on the one or more characteristics, setting, by the processor, one or more architectural parameters to constrain topologies of hidden layers used by an automated model generation process;
comparing the one or more characteristics to rules that map data set characteristics to model topology parameters;
executing, by the processor, the automated model generation process to generate a plurality of models, wherein the plurality of models comprises a model including data representative of a neural network; and
updating the rules based on ch

Assignees

• Sparkcognition Inc

Inventors

• Mcdonnell Tyler S
• Andoni Sari
• Choi Junhwan
• Goode Jimmie
• Lan Yiyun
• Moore Keith D
• Sellers Gavin

Classifications

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title