Layerwise Multi-Objective Neural Architecture Search for Optimization of Machine-Learned Models

Patent metadata

Field	Value
Publication number	US-2025238683-A1
Application number	US-202418596110-A
Country	US
Kind code	A1
Filing date	Mar 5, 2024
Priority date	Jan 23, 2024
Publication date	Jul 24, 2025
Grant date	—

Abstract

A search option can be selected for each model layer N of a plurality of model layers M. based on the model layer N, the one or more search options can be used to construct one or more candidate model layers for a model layer N+1 of the plurality of model layers. The one or more candidate model layers are respectively associated with one or more cost metrics. A cost metric is indicative of a cost associated with inclusion of the candidate model layer in an optimized machine-learned model. An optimized machine-learned model comprising M model layers can be constructed based on a cost function. The cost function maximizes an accuracy of the optimized machine-learned model subject to a sum of the cost metrics associated with each candidate model layer included in the optimized machine-learned model being less than a maximum cost.

First claim

What is claimed is:

1 . A computing system for layer-wise neural architecture search with polynomial complexity to combinatorically construct an optimized machine-learned model, comprising:
one or more processors; one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the participant computing device to perform operations, the operations comprising:
iteratively constructing, for each model layer of a plurality of model layers, one or more candidate model layers;
determining a cost metric for each of the candidate model layers, wherein a cost metric is indicative of a cost associated with inclusion of the candidate model layer in an optimized machine-learned model;
for each model layer, grouping the one or more respective candidate model layers into one or more candidate layer clusters, wherein each candidate layer cluster is associated with a range of cost metrics;
filtering at least one candidate model layer based the cost metric associated with the candidate model layer being greater than a threshold cost; and
constructing an optimized machine-learned model comprising a candidate model layer for each of the plurality of layers based on a cost function, wherein the cost function maximizes a performance metric of the optimized machine-learned model subject to a sum of the cost metrics associated with each candidate model layer included in the optimized machine-learned model being less than a maximum cost.

2 . The computing system of claim 1 , wherein the cost associated with selection of the candidate layer comprises one or more constraints, comprising:
a size of the candidate layer; a degree of energy consumption associated with the candidate layer; or an inference latency associated with the candidate layer.

3 . The computing system of claim 1 , wherein the operations further comprise:
determining, by the computing system, that the performance metric for the optimized machine-learned model is less than a threshold degree of performance; and constructing, by the computing system, a second optimized machine-learned model comprising a candidate model layer for each of the plurality of layers based on the cost function,, wherein the second cost function maximizes the performance metric of the optimized machine-learned model subject to a sum of the cost metrics associated with each candidate model layer included in the optimized machine-learned model being less than a second maximum cost greater than the maximum cost.

4 . A computer-implemented method to implement layerwise optimization of machine-learned models, the method comprising:
for each model layer N of a plurality of model layers M:
selecting, by a computing system comprising one or more computing devices, one or more layer search options from a plurality of layer search options;
based on the model layer N, using, by the computing system, the one or more search options to construct one or more candidate model layers for a model layer N+1 of the plurality of model layers, wherein the one or more candidate model layers are respectively associated with one or more cost metrics, wherein a cost metric is indicative of a cost associated with inclusion of the candidate model layer in an optimized machine-learned model; and
constructing, by the computing system, an optimized machine-learned model comprising M model layers based on a cost function, wherein the cost function maximizes a performance metric of the optimized machine-learned model subject to a sum of the cost metrics associated with each candidate model layer included in the optimized machine-learned model being less than a maximum cost.

5 . The computer-implemented method of claim 4 , wherein the cost associated with selection of the candidate layer comprises one or more constraints, comprising:
a size of the candidate layer; a degree of energy consumption associated with the candidate layer; or an inference latency associated with the candidate layer.

6 . The computer-implemented method of claim 4 , wherein the one or more candidate layers comprises a plurality of candidate layers respectively associated with a plurality of cost metrics, wherein each of the plurality of cost metrics is different, and wherein each candidate layer represents an optimal layer for a respectively associated cost metric; and
wherein using the one or more search options to identify one or more candidate layers further comprises: grouping, by the computing system, the plurality of candidate layers into a plurality of candidate layer clusters, wherein each candidate layer cluster is associated with a range of cost metrics.

7 . The computer-implemented method of claim 4 , wherein constructing the optimized machine-learned model comprises, for each layer of the optimized machine-learned model:
determining, by the computing system, a candidate layer cluster of the plurality of candidate layer clusters for the layer based on the cost function and the range of cost metrics associated with the candidate layer cluster; and selecting, by the computing system, a candidate layer from the candidate layer cluster based on the cost function and the cost metrics associated with one or more layers selected prior to the candidate layer.

8 . The computer-implemented method of claim 4 , wherein grouping the plurality of candidate layers into the plurality of candidate layer clusters further comprises storing, by the computing system, layer selection information indicative of the plurality of candidate layer clusters and each of the plurality of candidate layers; and
wherein determining the candidate layer cluster of the plurality of candidate layer clusters comprises determining, by the computing system, the candidate layer cluster of the plurality of candidate layer clusters for the layer based on the cost function and layer selection information.

9 . The computer-implemented method of claim 4 , wherein constructing the optimized machine-learned model based on the cost function comprises, for a model layer N of the plurality of model layers M:
determining, by the computing system for a candidate model layer for the model layer N, that the cost metrics associated with each candidate model layer constructed for the model layer N+1 based on the candidate model layer are greater than a maximum cost; and filtering, by the computing system, the candidate model layer from inclusion in the optimized machine-learned model.

10 . The computer-implemented method of claim 4 , wherein the method further comprises:
determining, by the computing system, that the performance metric for the optimized machine-learned model is less than a threshold degree of performance; and constructing, by the computing system, a second optimized machine-learned model comprising M model layers based on a second cost function, wherein the second cost function maximizes an accuracy of the optimized machine-learned model subject to a sum of the cost metrics associated with each candidate model layer included in the optimized machine-learned model being less than a second maximum cost greater than the maximum cost.

11 . The computer-implemented method of claim 4 , wherein, prior to selecting the one or more search options from a plurality of layer search options, the method comprises receiving, by the computing system, an optimization request indicative of a quantity of layers M and the maximum cost.

12 . One or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the participant computing device to perform operations, the operations comprisin

Assignees

• Google Llc

Inventors

• Fan Yicheng
• Kumar Keshav
• Tomkins Andrew
• Alon Dana
• Vee Erik Nathan
• Ravikumar Shanmugasundaram
• Shen Jingyue
• Peng Daiyi

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0985 Primary

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title