Systems and methods for optimization of a data model network architecture for target deployment

Patent metadata

Field	Value
Publication number	US-2019156178-A1
Application number	US-201816198321-A
Country	US
Kind code	A1
Filing date	Nov 21, 2018
Priority date	Nov 22, 2017
Publication date	May 23, 2019
Grant date	—

Abstract

Systems and methods are provided for selecting an optimized data model architecture subject to resource constraints. One or more resource constraints for target deployment are identified, and random model architectures are generated from a set of model architecture production rules subject to the one or more resource constraints. Each random model architecture is defined by randomly chosen values for one or more meta parameters and one or more layer parameters. One or more of the random model architectures are adaptively refined to improve performance relative to a metric, and the refined model architecture with the best performance relative to the metric is selected.

First claim

1 . A method of selecting an optimized data model architecture executed by a computing device equipped with a processor and a memory operatively coupled to the processor, comprising:
receiving, with the computing device, one or more resource constraints for a target deployment; generating, via the computing device, a plurality of random model architectures from a set of model architecture production rules subject to the one or more resource constraints, each random model architecture defined by randomly chosen values for one or more meta parameters and one or more layer parameters; adaptively refining, via the computing device, one or more of the plurality of random model architectures to improve performance relative to a metric; and selecting, via the computing device, the refined model architecture with the best performance relative to the metric.

2 . The method of claim 1 , wherein generating the plurality of random model architectures subject to the one more resource constraints comprises:
determining, via the computing device, for each random model architecture generated, whether the random model architecture satisfies the one or more resource constraints; and discarding, via the computing device, any of the generated random model architectures that fail to satisfy the one or more resource constraints prior to the adaptive refining of the one or more of the plurality of random model architectures.

3 . The method of claim 1 wherein adaptively refining the random model architectures is performed subject to the one or more resource constraints.

4 . The method of claim 1 wherein the one or more resource constraints include parameter count, runtime memory or number of floating point operations per second.

5 . The method of claim 1 wherein adaptively refining the random model architectures includes back-propagation of the random model architecture only to partial convergence.

6 . The method of claim 1 wherein adaptively refining the random model architectures includes iteratively adjusting one of the one or more layer parameters.

7 . The method of claim 1 wherein adaptively refining the random model architectures includes iteratively adjusting one of the one or more meta parameters.

8 . The method of claim 1 further comprising:
selecting, via the computing device, one or more of the plurality of random model architectures for adaptive refinement according to performance relative to a metric.

9 . The method of claim 8 , further comprising:
determining, via the computing device, performance relative to the metric including back-propagating the plurality of random model architectures only to partial convergence.

10 . The method of claim 1 wherein adaptively refining one or more of the plurality of random model architectures includes refining at least two of the plurality of random model architectures in parallel and selecting the random model architecture with the best performance relative to a metric for further refinement.

11 . The method of claim 1 wherein the one or more meta parameters include a number of layers or an ordering of layers.

12 . The method of claim 1 wherein the one or more layer parameters include a convolutional filter size, a number of filters, a stride length, or a padding length.

13 . The method of claim 1 wherein each of the plurality of model architectures includes a convolutional neural network.

14 . The method of claim 1 , wherein each of the plurality of random model architectures includes at least one composite layer.

15 . A non-transitory medium holding computer-executable instructions for selecting an optimized data model architecture, the instructions when executed causing at least one computing device to:
receive one or more resource constraints for a target deployment; generate a plurality of random model architectures from a set of model architecture production rules subject to the one or more resource constraints, each random model architecture defined by randomly chosen values for one or more meta parameters and one or more layer parameters; adaptively refine one or more of the plurality of random model architectures to improve performance relative to a metric; and select the refined model architecture with the best performance relative to the metric.

16 . The medium of claim 15 , wherein generating the plurality of random model architectures subject to the one more resource constraints comprises:
determining, via the computing device, for each random model architecture generated, whether the random model architecture satisfies the one or more resource constraints; and discarding, via the computing device, any of the generated random model architectures that fail to satisfy the one or more resource constraints prior to the adaptive refining of the one or more of the plurality of random model architectures.

17 . The medium of claim 15 wherein adaptively refining the random model architectures is performed subject to the one or more resource constraints.

18 . The medium of claim 15 wherein the one or more resource constraints include parameter count, runtime memory or number of floating point operations per second.

19 . The medium of claim 15 wherein adaptively refining the random model architectures includes back-propagation of the random model architecture only to partial convergence.

20 . The medium of claim 15 wherein adaptively refining the random model architectures includes iteratively adjusting one of the one or more layer parameters.

21 . The medium of claim 15 wherein adaptively refining the random model architectures includes iteratively adjusting one of the one or more meta parameters.

22 . The medium of claim 15 wherein the instructions when executed further cause the at least one computing device to:
select one or more of the plurality of random model architectures for adaptive refinement according to performance relative to a metric.

23 . The medium of claim 22 , wherein the instructions when executed further cause the at least one computing device to:
determine performance relative to the metric including back-propagating the plurality of random model architectures only to partial convergence.

24 . The medium of claim 15 wherein adaptively refining one or more of the plurality of random model architectures includes refining at least two of the plurality of random model architectures in parallel and selecting the random model architecture with the best performance relative to a metric for further refinement.

25 . The medium of claim 15 wherein the one or more meta parameters include a number of layers or an ordering of layers.

26 . The medium of claim 15 wherein the one or more layer parameters include a convolutional filter size, a number of filters, a stride length, or a padding length.

27 . The medium of claim 15 wherein each of the plurality of model architectures includes a convolutional neural network.

28 . The medium of claim 15 , wherein each of the plurality of random model architectures includes at least one composite layer.

29 . A system for selecting an optimized data model architecture, comprising:
a computing device including a processor and a memory operatively coupled to the processor, the memory having instructions stored therein that when executed by the processor cause the computing device to:

Assignees

• Massachusetts Inst Technology

Inventors

• Thornton Jason R
• Skelly Luke
• Chan Michael
• Duarte Ronald
• Scarafoni Daniel

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N5/046

  Forward inferencing; Production systems · CPC title