Weakly supervised anomaly detection and segmentation in images

Patent metadata

Field	Value
Publication number	US-10672115-B2
Application number	US-201716075167-A
Country	US
Kind code	B2
Filing date	Dec 6, 2017
Priority date	Dec 6, 2016
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

Systems and methods are disclosed for processing an image to detect anomalous pixels. An image classification is received from a trained convolutional neural network (CNN) for an input image with a positive classification being defined to represent detection of an anomaly in the image and a negative classification being defined to represent absence of an anomaly. A backward propagation analysis of the input image for each layer of the CNN generates an attention mapping that includes a positive attention map and a negative attention map. A positive mask is generated based on intensity thresholds of the positive attention map and a negative mask is generated based on intensity thresholds of the negative attention map. An image of segmented anomalous pixels is generated based on an aggregation of the positive mask and the negative mask.

First claim

What is claimed is:

1. A computer-implemented method for processing an image of an object, comprising:
receiving an image classification from a trained convolutional neural network (CNN) for an input image of the object, wherein the image classification is a positive classification that corresponds to detection of an anomaly in the input image and the image classification is a negative classification that corresponds to absence of an anomaly in the input image;
performing a backward propagation analysis of the input image for each layer of the CNN to generate an attention mapping, wherein the attention mapping includes a positive attention map generated from a positive signal in the backward propagation and a negative attention map generated from a negative signal in the backward propagation;
generating a positive mask based on intensity thresholds of the positive attention map and a negative mask based on intensity thresholds of the negative attention map;
generating an aggregated mask by aggregating the positive mask and the negative mask; and
determining an image of segmented anomalous pixels based on a segmentation of the aggregated mask.

2. The method of claim 1 , further comprising:
encoding the positive attention map with the location of the anomalous pixels;
determining a first threshold and a second threshold for an intensity distribution of the positive attention map; and
performing a color model estimation of the positive attention map pixels, wherein pixels with an intensity exceeding the first threshold are converted to a first color, pixels with an intensity below the second threshold are converted to a second color, and pixels with an intensity between the first threshold and the second threshold are converted to a third color.

3. The method of claim 2 , wherein the first color represents foreground pixels, the second color represents background pixels, and the third color represent pixels not used for segmentation of anomalies.

4. The method of claim 3 , further comprising:
performing a graph cut segmentation on the color model estimation using the input image to generate the image of segmented anomalous pixels.

5. The method of claim 1 , wherein performing a backward propagation analysis comprises:
computing a gradient derivative value for each pixel decision at each convolution layer.

6. The method of claim 1 , wherein the input image comprises patches, further comprising:
performing a classification on each patch of the input image, and
performing the backward propagation on a patch in response to detecting a positive classification for the patch.

7. The method of claim 6 , further comprising:
dividing the input image into patches in response to detecting a positive classification.

8. A system, comprising:
at least one memory storing computer-executable instructions; and
at least one processor configured to access the at least one memory and execute the instructions to:
receive an image classification from a trained convolutional neural network (CNN) for an input image of an object, wherein the image classification is a positive classification that corresponds to detection of an anomaly in the input image and the image classification is a negative classification that corresponds to absence of an anomaly in the input image;
perform a backward propagation analysis of the input image for each layer of the CNN to generate an attention mapping, wherein the attention mapping includes a positive attention map generated from a positive signal in the backward propagation and a negative attention map generated from a negative signal in the backward propagation;
generate a positive mask based on intensity thresholds of the positive attention map and a negative mask based on intensity thresholds of the negative attention map;
generate an aggregated mask by aggregating the positive mask and the negative mask; and
determine an image of segmented anomalous pixels based on a segmentation of the aggregated mask.

9. The system of claim 8 , wherein the processor is configured to access the at least one memory and execute the instructions to:
encode the positive attention map with the location of the anomalous pixels;
determine a first threshold and a second threshold for an intensity distribution of the positive attention mapping; and
perform a color model estimation of the positive attention map pixels, wherein pixels with an intensity exceeding the first threshold are converted to a first color, pixels with an intensity below the second threshold are converted to a second color, and pixels with an intensity between the first threshold and the second threshold are converted to a third color.

10. The system of claim 9 , wherein the first color represents foreground pixels, the second color represents background pixels, and the third color represent pixels not used for segmentation of anomalies.

11. The system of claim 10 , wherein the processor is configured to access the at least one memory and execute the instructions to:
perform a graph cut segmentation on the color model estimation using the input image to generate the image of segmented anomalous pixels.

12. The system of claim 8 , wherein the processor is configured to access the at least one memory and execute the instructions to:
compute a gradient derivative value for each pixel decision at each convolution layer when performing a backward propagation.

13. The system of claim 8 , wherein the processor is configured to access the at least one memory and execute the instructions to:
perform a classification on each patch of the input image, and
perform the backward propagation on a patch in response to detecting a positive classification for the patch.

14. The system of claim 13 , wherein the processor is configured to access the at least one memory and execute the instructions to:
divide the input image into patches in response to detecting a positive classification.

Assignees

• Siemens Energy Inc
• Siemens Corp

Inventors

• Panda Rameswar
• Wu Ziyan
• Innanje Arun
• Nair Ramesh
• Chang Ti-Chiun
• Ernst Jan

Classifications

• G06T7/0004 Primary

  Industrial image inspection · CPC title

• G06N3/0454

  Physics · mapped topic

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2207/20081

  Training; Learning · CPC title