Method for co-design of hardware and neural network architectures using coarse-to-fine search, two-phased block distillation and neural hardware predictor

Patent metadata

Field	Value
Publication number	US-12112112-B2
Application number	US-202017095937-A
Country	US
Kind code	B2
Filing date	Nov 12, 2020
Priority date	Nov 12, 2020
Publication date	Oct 8, 2024
Grant date	Oct 8, 2024

Abstract

Methods, systems, and apparatus for combined or separate implementation of coarse-to-fine neural architecture search (NAS), two-phase block NAS, variable hardware prediction, and differential hardware design are provided and described. A variable predictor is trained, as described herein. Then, a controller or policy may be used to iteratively modify a neural network architecture along dimensions formed by neural network architecture parameters. The modification is applied to blocks (e.g., subnetworks) within the neural network architecture. In each iteration, the remainder of the neural network architecture parameters are modified and learned with a differential NAS method. The training process is performed with two-phase block NAS and incorporates a variable hardware predictor to predict power, performance, and area (PPA) parameters. The hardware parameters may be learned as well using the variable hardware predictor.

First claim

What is claimed is:

1. A method comprising:
identifying a plurality of dimensions of a neural architecture search into a first group of dimensions including a coarse dimension and a second group of dimensions including a fine dimension;
selecting a value of the coarse dimension during a coarse search phase of a neural architecture search using a controller based search algorithm, wherein the controller based search algorithm for the coarse search phase includes iteratively altering the value of the coarse dimension using a policy function;
identifying a search space corresponding to the fine dimension from the second group of dimensions for a fine search phase of the neural architecture search based at least in part on the value of the coarse dimension selected during the coarse search phase of the neural architecture search;
selecting a value for the fine dimension during the fine search phase of the neural architecture search using a differential search algorithm, wherein the differential search algorithm is based on a loss function; and
generating a neural network architecture based on the value of the coarse dimension and the value of the fine dimension.

2. The method of claim 1 , wherein:
the coarse dimension comprises a number of layers, a layer type, a layer connectivity, or any combination thereof.

3. The method of claim 1 , wherein:
the fine dimension comprises a layer size, a number of channels, a filter size, a cell size, a gate size, an activation function, or any combination thereof.

4. The method of claim 1 , wherein:
the controller based search algorithm comprises a reinforcement learning algorithm or an evolutionary search algorithm.

5. The method of claim 1 , further comprising:
configuring a block of a neural network based on the neural architecture search;
computing a first loss function based on an intermediate output of the block;
updating the neural architecture search for the block based on the first loss function;
computing a second loss function using a final output of the neural network after updating neural architecture search based on the first loss function; and
updating the neural architecture search for the neural network based on the second loss function.

6. The method of claim 1 , further comprising:
identifying neural architecture parameters including the value of the coarse dimension and the value of the fine dimension;
identifying a set of hardware parameters;
predicting power, performance, and area (PPA) parameters using a variable hardware predictor;
iteratively updating the neural architecture parameters and the hardware parameters based on the predicted PPA parameters; and
generating a hardware design based on the updated hardware parameters.

7. The method of claim 6 , further comprising:
selecting the hardware parameters using a neural network trained using a differential training algorithm.

8. An apparatus comprising a processor and a memory storing instructions and in electronic communication with the processor, the processor being configured to execute the instructions to:
identify a plurality of dimensions of a neural architecture search into a first group of dimensions including a coarse dimension and a second group of dimensions including a fine dimension;
select a value of the coarse dimension during a coarse search phase of a neural architecture search using a controller based search algorithm, wherein the controller based search algorithm for the coarse search phase includes iteratively altering the value of the coarse dimension using a policy function;
identify a search space corresponding to the fine dimension from the second group of dimensions for a fine search phase of the neural architecture search based at least in part on the value of the coarse dimension selected during the coarse search phase of the neural architecture search;
select a value for the fine dimension during the fine search phase of the neural architecture search using a differential search algorithm, wherein the differential search algorithm is based on a loss function; and
generate a neural network architecture based on the value of the coarse dimension and the value of the fine dimension.

9. The apparatus of claim 8 , wherein:
the coarse dimension comprises a number of layers, a layer type, a layer connectivity, or any combination thereof.

10. The apparatus of claim 8 , wherein:
the fine dimension comprises a layer size, a number of channels, a filter size, a cell size, a gate size, an activation function, or any combination thereof.

11. The apparatus of claim 8 , wherein:
the controller based search algorithm comprises a reinforcement learning algorithm or an evolutionary search algorithm.

12. The apparatus of claim 8 , the processor being further configured to execute the instructions to:
configure a block of a neural network based on the neural architecture search;
compute a first loss function based on an intermediate output of the block;
update the neural architecture search for the block based on the first loss function;
compute a second loss function using a final output of the neural network after updating neural architecture search based on the first loss function; and
update the neural architecture search for the neural network based on the second loss function.

13. The apparatus of claim 8 , the processor being further configured to execute the instructions to:
identify neural architecture parameters including the value of the coarse dimension and the value of the fine dimension;
identify a set of hardware parameters;
predict power, performance, and area (PPA) parameters using a variable hardware predictor;
iteratively update the neural architecture parameters and the hardware parameters based on the predicted PPA parameters; and
generate a hardware design based on the updated hardware parameters.

14. The apparatus of claim 13 , the processor being further configured to execute the instructions to:
select the hardware parameters using a neural network trained using a differential training algorithm.

15. A non-transitory computer readable medium storing code, the code comprising instructions executable by a processor to:
identifying a plurality of dimensions of a neural architecture search into a first group of dimensions including a coarse dimension and a second group of dimensions including a fine dimension;
select a value of the coarse dimension during a coarse search phase of a neural architecture search using a controller based search algorithm, wherein the controller based search algorithm for the coarse search phase includes iteratively altering the value of the coarse dimension using a policy function;
identify a search space corresponding to the fine dimension from the second group of dimensions for a fine search phase of the neural architecture search based at least in part on the value of the coarse dimension selected during the coarse search phase of the neural architecture search;
select a value for the fine dimension during the fine search phase of the neural architecture search using a differential search algorithm, wherein the differential search algorithm is based on a loss function; and
generate a neural network architecture based on the value of the coarse dimension and the value of the fine dimension.

16. The non-transitory computer readable medium of claim 15 , wherein:
the coarse dimension comprises a number of layers, a layer type, a layer connectivity, or any combination thereof.

17. The non-transitory computer readable medium of claim 15 , wherei

Assignees

• Samsung Electronics Co Ltd

Inventors

• Zehngut Niv
• Ben-Dror Amir
• Artyomov Evgeny
• Dinerstein Michael
• Jevnisek Roy

Classifications

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title