Device and method for searching neural network architecture using supernet

Patent metadata

Field	Value
Publication number	US-2025028959-A1
Application number	US-202318455183-A
Country	US
Kind code	A1
Filing date	Aug 24, 2023
Priority date	Jul 18, 2023
Publication date	Jan 23, 2025
Grant date	—

Abstract

A method for searching a neural network architecture using supernets comprises the steps of: (a) searching for subnets that can be extracted from a set search space; (b) counting the number of non-linear activation functions included in each subnet for each of the searched subnets; (c) grouping the searched subnets based on the counted number of non-linear activation functions; (d) assigning the subnet groups to multiple supernets; (e) searching for a neural network having an optimal architecture based on operation blocks of the subnet groups assigned to each of the multiple supernets.

First claim

What is claimed is:

1 . A method for searching a neural network architecture, the method comprising the steps of:
(a) searching for subnets that can be extracted from a set search space; (b) counting the number of non-linear activation functions included in each subnet for each of the searched subnets; (c) grouping the searched subnets based on the counted number of non-linear activation functions; (d) assigning the subnet groups to multiple supernets; (e) searching for a neural network having an optimal architecture based on operation blocks of the subnet groups assigned to each of the multiple supernets.

2 . The method for searching a neural network architecture according to claim 1 ,
wherein the step (c) includes grouping subnets with the same number of non-linear activation functions into the same group.

3 . The method for searching a neural network architecture according to claim 1 ,
wherein the step (b) includes counting the number of non-linear activation functions by counting the number of operation blocks set to use the non-linear activation function among the operation blocks included in the extracted subnets.

4 . The method for searching a neural network architecture according to claim 3 ,
wherein when the extracted subnet is a neural network having a parallel architecture, the number of non-linear activation functions is counted for each path of the parallel architecture, and the number of non-linear activation functions of the path having the largest number of non-linear activation functions among a plurality of paths is determined as the number of non-linear activation functions of the corresponding subnet.

5 . The method for searching a neural network architecture according to claim 1 ,
wherein in step (d), subnets belonging to the same group are assigned to the same supernet.

6 . The method for searching a neural network architecture according to claim 1 ,
wherein the number of supernets is determined for each subnet group based on the variance value of the distribution after obtaining the distribution of the number of subnets included in the group.

7 . The method for searching a neural network architecture according to claim 1 ,
wherein in order to determine the number of supernets, groups including subnets greater than a preset critical value are searched among the plurality of groups grouped in step (c), and the number of groups including subnets greater than the preset critical value is determined as the number of supernets.

8 . The method for searching a neural network architecture according to claim 1 ,
wherein in step (e), each of the multiple supernets extracts subnets corresponding to the number of non-linear activation functions associated with the group assigned to the corresponding supernet, thereby searching for a neural network having an optimal architecture.

9 . A device for searching a neural network architecture, the device including:
a processor; and at least one memory connected to the processor, wherein the processor executes the steps of: (a) searching for subnets that can be extracted from a set search space; (b) counting the number of non-linear activation functions included in each subnet for each of the searched subnets; (c) grouping the searched subnets based on the counted number of non-linear activation functions; (d) assigning the subnet groups to multiple supernets; (e) searching for a neural network having an optimal architecture based on operation blocks of the subnet groups assigned to each of the multiple supernets.

10 . The device for searching a neural network architecture according to claim 9 ,
wherein the step (c) includes grouping subnets with the same number of non-linear activation functions into the same group.

11 . The device for searching a neural network architecture according to claim 9 ,
wherein the step (b) includes counting the number of non-linear activation functions by counting the number of operation blocks set to use the non-linear activation function among the operation blocks included in the extracted subnets.

12 . The device for searching a neural network architecture according to claim 11 ,
wherein when the extracted subnet is a neural network having a parallel architecture, the number of non-linear activation functions is counted for each path of the parallel architecture, and the number of non-linear activation functions of the path having the largest number of non-linear activation functions among a plurality of paths is determined as the number of non-linear activation functions of the corresponding subnet.

13 . The device for searching a neural network architecture according to claim 9 ,
wherein in step (d), subnets belonging to the same group are assigned to the same supernet.

14 . The device for searching a neural network architecture according to claim 9 ,
wherein the number of supernets is determined for each subnet group based on the variance value of the distribution after obtaining the distribution of the number of subnets included in the group.

15 . The device for searching a neural network architecture according to claim 9 ,
wherein in order to determine the number of supernets, groups including subnets greater than a preset critical value are searched among the plurality of groups grouped in step (c), and the number of groups including subnets greater than the preset critical value is determined as the number of supernets.

16 . The device for searching a neural network architecture according to claim 9 ,
wherein in step (e), each of the multiple supernets extracts subnets corresponding to the number of non-linear activation functions associated with the group assigned to the corresponding supernet, thereby searching for a neural network having an optimal architecture.

Assignees

• Uif Univ Industry Foundation Yonsei Univ

Inventors

• Ham Bum Sub
• Oh Young Min

Classifications

• G06N3/048 Primary

  Activation functions · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/082 Primary

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title