System and method for learning the structure of deep convolutional neural networks

Patent metadata

Field	Value
Publication number	US-11010658-B2
Application number	US-201715853403-A
Country	US
Kind code	B2
Filing date	Dec 22, 2017
Priority date	Dec 22, 2017
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

A recursive method and apparatus produce a deep convolution neural network (CNN). The method iteratively processes an input directed acyclic graph (DAG) representing an initial CNN, a set of nodes, a set of exogenous nodes, and a resolution based on the CNN. An iteration for a node may include recursively performing the iteration upon each node in a descendant node set to create a descendant DAG, and upon each node in ancestor node sets to create ancestor DAGs, the ancestor node sets being a remainder of nodes in the temporary DAG after removing nodes of the descendent node set. The descendant and ancestor DAGs are merged, and a latent layer is created that includes a latent node for each ancestor node set. Each latent node is set to be a parent of sets of parentless nodes in a combined descendant DAG and ancestors DAGs before returning.

First claim

What is claimed is:

1. An apparatus for producing a deep convolution neural network (CNN), the apparatus comprising: memory and processing circuitry coupled to the memory, the processing circuitry to:
iteratively process an input directed acyclic graph (DAG), a set of nodes, a set of exogenous nodes, and a resolution based on the CNN, an iteration for a node in the set of nodes of the CNN including the processing circuitry to further:
increase a resolution of the DAG by creating a temporary DAG used for a current recursive iteration based on the DAG and the resolution;
recursively perform the iteration upon each node in a descendant node set to create a descendant DAG, the descendent node set being autonomous nodes having a lowest topological order;
recursively perform the iteration upon each node in ancestor node sets to create ancestor DAGs, the ancestor node sets being the remainder of nodes in the temporary DAG after nodes of the descendent node set are removed, wherein the descendant node set is a common child of the ancestor node sets;
merge the descendant DAG and the ancestor DAGs into a processed DAG;
create a latent layer in the processed DAG that includes a latent node for each ancestor node set for the current resolution;
set each latent node to be a parent of sets of parentless nodes in a combined descendant DAG and ancestors DAGs; and
return the processed DAG.

2. The apparatus of claim 1 , wherein the increasing of the resolution of the DAG comprises having the processing circuitry to further:
for each pair of connected nodes in the temporary DAG:
disconnect a pair of the connected nodes when the connected nodes are independent of one another; and
direct node edges of the disconnected nodes to a common neighbor node when the common neighbor node is not independent of the disconnected nodes.

3. The apparatus of claim 2 , wherein the determination of when the pair of the connected nodes are independent is done by a statistical test and relates to a statistical dependency of node activations.

4. The apparatus of claim 3 , wherein the statistical test is a conditional independence (CI) test (CIT).

5. The apparatus of claim 4 , wherein the CIT is a partial correlation test or a conditional mutual information test.

6. The apparatus of claim 5 , wherein the CIT is a binary test yielding a binary value of zero or one of a form:

CI
⁡

(

W
X

⁡

(
i
)

,

W
Y

⁡

(
i
)

|

W
Z

⁡

(
i
)

)

=

CI
⁡

(

𝒟
X

⁡

(
i
)

,

𝒟
Y

⁡

(
i
)

|

𝒟
Z

⁡

(
i
)

)

=

meas
⁡

(

𝒟
X

⁡

(
i
)

,

𝒟
Y

⁡

(
i
)

|

𝒟
Z

⁡

(
i
)

)

>
γ

where:
W X (i), W Y (i), and W Z (i) are activations selected by a window from feature maps X, Y, and Z at location i;
X (i) and Y (i)) are input variables;
Z (i) is a condition set of variables;
meas( ) evaluates a level of correlation; and
γ is a

Assignees

• Intel Corp

Inventors

• Koren Guy
• Yehezkel Rohekar Raanan Yonatan
• Nisimov Shami
• Novik Gal

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0464 Primary

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title