Hardware-aware efficient neural network design system having differentiable neural architecture search

Patent metadata

Field	Value
Publication number	US-11748615-B1
Application number	US-201916704971-A
Country	US
Kind code	B1
Filing date	Dec 5, 2019
Priority date	Dec 6, 2018
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

Computer implemented systems are described that implement a differentiable neural architecture search (DNAS) engine executing on one or more processors. The DNAS engine is configured with a stochastic super net defining a layer-wise search space having a plurality of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture. Further, the DNAS engine is configured to process training data to train weights for the operators in the stochastic super net based on a loss function representing a latency of the respective operator on a target platform, and to select a set of candidate neural network architectures from the trained stochastic super net. The DNAS engine may, for example, be configured to train the stochastic super net by traversing the layer-wise search space using gradient-based optimization of network architecture distribution.

First claim

The invention claimed is:

1. A computer-implemented system comprising:
a differentiable neural architecture search (DNAS) engine executing on one or more processors, wherein the DNAS engine is configured with a stochastic super net defining a layer-wise search space having a plurality of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture,
wherein the DNAS engine is configured to train the stochastic super net by processing training data to train weights for the operators in the stochastic super net, wherein the DNAS engine is configured to train a weight for a respective operator based on a loss function representing a latency of the respective operator on a target platform, wherein the latency of the respective operator is based on an estimated runtime of the respective operator on the target platform that is independent of runtimes of other operators on the target platform, and
wherein the DNAS engine is configured to select a set of candidate neural network architectures from the trained stochastic super net.

2. The system of claim 1 , wherein the DNAS engine is configured to train the stochastic super net by traversing the layer-wise search space using gradient-based optimization of network architecture distribution.

3. The system of claim 1 ,
wherein the stochastic super net comprises each of the candidate layers having a plurality of parallel candidate blocks, and
wherein the DNAS engine is configured to train the stochastic super net by sampling each of the candidate layers to select and execute one of the candidate blocks from each of the candidate layers.

4. The system of claim 1 ,
wherein the layer-wise search space defines a set of input and output dimensions of image data for each of the candidate layers, and
wherein each of the candidate layers is associated with a corresponding image block type.

5. The system of claim 1 , wherein one or more of the candidate layers is associated with a corresponding number of output filters.

6. The system of claim 1 , wherein, to train the stochastic super net, the DNAS engine is configured to access a latency lookup table that defines the estimated runtime of the respective operator on the target platform.

7. A method comprising:
constructing a stochastic super net defining a layer-wise search space having a number of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture;
training the stochastic super net by processing training data to train weights for the operators in the stochastic super net, wherein training the stochastic super net includes training a weight for a respective operator based on a loss function associated with a latency of the respective operator on a target platform, wherein the latency of the respective operator is based on an estimated runtime of the respective operator on the target platform that is independent of runtimes of other operators on the target platform; and
selecting a set of candidate neural network architectures from the trained stochastic super net.

8. The method of claim 7 , wherein training the stochastic super net comprises traversing the layer-wise search space using a gradient-based optimization of network architecture distribution.

9. The method of claim 7 ,
wherein constructing the stochastic super net comprises constructing each of the candidate layers to have a plurality of parallel candidate blocks, and
wherein training the stochastic super net comprises sampling each of the candidate layers to select and execute, with the training data, one of the candidate blocks from each of the candidate layers.

10. The method of claim 7 ,
wherein constructing the stochastic super net comprises constructing the layer-wise search space to define a set of input and output dimensions of image data for each of the candidate layers and associating each of the candidate layers with a corresponding image block type.

11. The method of claim 7 , wherein training the stochastic super net comprises accessing a latency lookup table that defines the estimated runtime of the respective operator on the target platform.

12. A computer-readable medium comprising instructions for causing one or more programmable processors to:
construct a stochastic super net defining a layer-wise search space having a number of candidate layers, each of the candidate layers specifying one or more operators for a neural network architecture;
train the stochastic super net by processing training data to train weights for the operators in the stochastic super net, wherein the instructions cause the one or more programmable processors to train a weight for a respective operator based on a loss function associated with a latency of the respective operator on a target platform, wherein the latency of the respective operator is based on an estimated runtime of the respective operator on the target platform that is independent of runtimes of other operators on the target platform; and
select a set of candidate neural network architectures from the trained stochastic super net.

13. The computer-readable medium of claim 12 , further comprising instructions to train the stochastic super net by traversing the layer-wise search space using a gradient-based optimization of network architecture distribution.

14. The computer-readable medium of claim 12 , further comprising instructions to:
construct the stochastic super net by constructing each of the candidate layers to have a plurality of parallel candidate blocks, and
train the stochastic super net by sampling each of the candidate layers to select and execute, with the training data, one of the candidate blocks from each of the candidate layers.

15. The computer-readable medium of claim 12 , further comprising instructions to
construct the stochastic super net by constructing the layer-wise search space to define a set of input and output dimensions of image data for each of the candidate layers and associating each of the candidate layers with a corresponding image block type.

16. The computer-readable medium of claim 12 , further comprising instructions to train the stochastic super net by accessing a latency lookup table that defines the estimated runtime of the respective operator on the target platform, and computing an overall latency of one or more candidate convolutional neural network models according to estimated runtimes for the operators in the stochastic super net.

Assignees

• Meta Platforms Inc

Inventors

• Wu Bichen
• Zhang Peizhao
• Vajda Peter
• Dai Xiaoliang
• Wang Yanghan
• Tian yuandong

Classifications

• G06N3/0985 Primary

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title