Image-based defects identification and semi-supervised localization

Patent metadata

Field	Value
Publication number	US-2021319546-A1
Application number	US-202016938812-A
Country	US
Kind code	A1
Filing date	Jul 24, 2020
Priority date	Apr 10, 2020
Publication date	Oct 14, 2021
Grant date	—

Abstract

A system for manufacturing defect classification is presented. The system includes a first neural network receiving a first data as input and generating a first output, a second neural network receiving a second data as input and generating a second output, wherein first neural network and the second neural network are trained independently from each other, and a fusion neural network receiving the first output and the second output and generating a classification. The first data and the second data do not have to be aligned. Hence, the system and method of this disclosure allows various type of data that are collected during manufacturing to be used in defect classification.

First claim

What is claimed is:

1 . A system for manufacturing defect classification, comprising:
a first neural network branch receiving a first data as input and generating a first output; a second neural network branch receiving a second data as input and generating a second output, wherein first neural network branch and the second neural network branch are trained independently from each other; and a fusion neural network receiving the first output and the second output and generating a classification.

2 . The system of claim 1 , wherein the first data and the second data are unaligned with respect to each other.

3 . The system of claim 1 , wherein one of the first data includes spectroscopy images.

4 . The system of claim 1 , wherein the second data includes microscopy images.

5 . The system of claim 1 , further comprising at least one of a channel attention module and a spatial attention module overlaid onto at least one of the first data and the second data, wherein the channel attention module and the spatial attention module are trained in a semi-supervised manner such that extra weight is put on select parts of the first data or the second data.

6 . The system of claim 5 , wherein each of the first neural network branch and the second neural network branch comprises a plurality of blocks, and the channel attention module and the spatial attention module are overlaid as a convolutional block attention module (CBAM) on each one of the plurality of blocks.

7 . The system of claim 1 , wherein the first data are Energy-Dispersive X-ray Spectroscopy (EDS) images, the first neural network branch further comprising a channel attention module to put additional weight to a channel of the first data, the channel corresponding a target element.

8 . The system of claim 1 , wherein the first data comprise multiple images having multiple channels.

9 . The system of claim 1 , wherein the fusion neural network comprises a convolutional block attention module having a spatial attention module and a channel attention module.

10 . A computer-implemented method for classification, comprising:
receiving a first output from a first neural network branch that takes first data as input; receiving a second output from a second neural network branch that takes second data as input, wherein the first neural network and the second neural network are trained independently of each other; and inputting the first output and the second output into a fusion neural network to generate a final classification.

11 . The computer-implemented method of claim 10 , wherein the first data and the second data are not aligned with respect to each other.

12 . The computer-implemented method of claim 10 , wherein the first data includes spectroscopy images.

13 . The computer-implemented method of claim 10 , wherein one of the second data includes microscopy images.

14 . The computer-implemented method of claim 10 , further comprising overlaying a spatial attention module onto at least one of the first data and the second data, wherein the channel attention module and the spatial attention module are trained in a semi-supervised manner to put on extra weight on select parts of the first data or the second data.

15 . The computer-implemented method of claim 14 , wherein the channel attention module and the spatial attention module are overlaid as a convolutional block attention module (CBAM) on a block in the first neural network or the second neural network.

16 . The computer-implemented method of claim 10 , wherein the first data are Energy-Dispersive X-ray Spectroscopy (EDS) images, the method further comprising applying a channel attention module the first neural network to put additional weight to a channel of the first data, the channel corresponding to a target element.

17 . The computer-implemented method of claim 10 , wherein the first data comprise multiple channels.

18 . The computer-implemented method of claim 10 , further comprising overlaying a convolutional block attention module (CBAM) onto the fusion neural network.

19 . The computer-implemented method of claim 1 , further comprising augmenting at least one of the first data and the second data to provide additional data points for training.

Assignees

• Samsung Display Co Ltd

Inventors

• Kang Yan
• Lee Janghwan
• Qu Shuhui
• Yao Jinghua
• Marapareddy Sai

Classifications

• G06V20/69

  Microscopic objects, e.g. biological cells or cellular parts · CPC title

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06F18/241

  relating to the classification model, e.g. parametric or non-parametric approaches · CPC title

• G06F18/251

  of input or preprocessed data · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title