Verification of classification decisions in convolutional neural networks

Patent metadata

Field	Value
Publication number	US-2022019870-A1
Application number	US-201917294746-A
Country	US
Kind code	A1
Filing date	Nov 12, 2019
Priority date	Nov 19, 2018
Publication date	Jan 20, 2022
Grant date	—

Abstract

In one aspect the invention relates to a computer-implemented method for providing a computer-implemented method for verifying a visual classification architecture of a convolutional neural network (CNN) and its decisions The method comprises to access (S1) a memory (MEM) with a convolutional neural network (CNN), being trained for a visual classification task into a set of target classes (tc); to use (S2) the convolutional neural network (CNN) for an input image (12) and after a forward pass of the convolutional neural network (CNN), in a backward pass: to apply (S3) a contrastive layer-wise relevance propagation algorithm (CLRP) or to apply (S4) a Bottom Up Attention pattern (BUAP), which is implicitly learned by the convolutional neural network (CNN) for providing (S5) a verification signal (vs).

First claim

1 . A computer-implemented method for verifying a visual classification architecture of a convolutional neural network, the method comprising:
accessing a memory with a convolutional neural network, being trained for a visual classification task into a set of target classes; and using the convolutional neural network for an input image and after a forward pass of the convolutional neural network, in a backward pass:
applying a contrastive layer-wise relevance propagation (CLRP) algorithm or
applying an implicitly learned Bottom Up Attention pattern, to verify a classification ability of the convolutional neural network
for providing a verification signal, wherein the CLRP algorithm comprises:
generating a first saliency map for each target class of the classification task by a backpropagation algorithm;
calculating a set of virtual classes for each target class, being opposite of the respective target class;
generating a second saliency map for the set of virtual classes by a backpropagation algorithm; and
computing the differences between the first saliency map and the second saliency map for computing a final saliency map.

2 . The method according to claim 1 , wherein the verification signal is provided as a saliency map for each feature on each layer of the convolutional neural network.

3 . The method according to claim 1 , wherein by applying the CLRP algorithm, class discriminative and instance-specific saliency maps are generated.

4 . The method according to claim 2 , wherein for applying an implicitly learned Bottom Up Attention pattern, a deconvolutional CNN algorithm, a gradient backpropagation algorithm or a layer-wise backpropagation algorithm are amended in order to generate saliency maps for features and not for classes.

5 . The method according to claim 1 , wherein calculating the virtual class for a specific target class is executed by:
defining any other of the set of target classes as virtual class, or by defining all other target classes of the set of target classes as virtual class, or by constructing the virtual class by generating an additional class and connecting it with a last layer using weights, wherein the weights are the inverted weights of the forward pass.

6 . The method according to claim 4 , wherein applying the Bottom Up Attention pattern comprises:
collecting and storing all features of the CNN, wherein a feature comprises all activations in a respective layer of the CNN for the input image; creating a saliency map for each of the features.

7 . The method according to claim 1 , wherein the visual classification task is a medical classification task in medical images in order to detect anomalies.

8 . The method according to claim 1 , wherein application of the convolutional neural network is only approved, if the provided verification signal is above a pre-configurable confidence threshold.

9 . The method according to claim 6 , wherein when applying a Bottom Up Attention pattern for generating a saliency map a guided backpropagation algorithm is used.

10 . The method according to claim 1 , wherein the generated saliency maps are post processed and/or may be refined and/or an averaging and/or a thresholding may be applied.

11 . A verification unit which is configured for verifying a visual classification architecture of a convolutional neural network, comprising:
a memory with the CNN, being trained for a visual classification task into a set of target classes; a processor which is configured for using the CNN and wherein the processor is configured after a forward pass of the CNN, in a backward pass: to apply a contrastive layer-wise relevance propagation (CLRP) algorithm or to apply a Bottom Up Attention pattern, which is implicitly learned by the CNN for generating a saliency map for each of the target classes, wherein the CLRP algorithm comprises:
generating a first saliency map for each target class of the classification task by a backpropagation algorithm;
calculating a set of virtual classes for each target class, being opposite of the respective target class;
generating a second saliency map for the set of virtual classes by a backpropagation algorithm; and
computing the differences between the first saliency map and the second saliency map for computing a final saliency map.

12 . A computer program product, comprising a computer readable hardware storage device having computer readable program code stored therein, said program code executable by a processor of a computer system to implement the method according to claim 1 , when the program elements are loaded into a memory of the computer.

13 . A computer-readable medium on which a convolutional neural network and program elements are stored that can be read and executed by a computer in order to perform steps of the method according to claim 1 , when the program elements are executed by the computer.

Assignees

• Siemens Ag

Inventors

• Gu Jindong

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/04 Primary

  Architecture, e.g. interconnection topology · CPC title