Active learning for defect classifier training

Patent metadata

Field	Value
Publication number	US-2019370955-A1
Application number	US-201916424431-A
Country	US
Kind code	A1
Filing date	May 28, 2019
Priority date	Jun 5, 2018
Publication date	Dec 5, 2019
Grant date	—

Abstract

Methods and systems for performing active learning for defect classifiers are provided. One system includes one or more computer subsystems configured for performing active learning for training a defect classifier. The active learning includes applying an acquisition function to data points for the specimen. The acquisition function selects one or more of the data points based on uncertainty estimations associated with the data points. The active learning also includes acquiring labels for the selected one or more data points and generating a set of labeled data that includes the selected one or more data points and the acquired labels. The computer subsystem(s) are also configured for training the defect classifier using the set of labeled data. The defect classifier is configured for classifying defects detected on the specimen using the images generated by the imaging subsystem.

First claim

What is claimed is:

1 . A system configured to perform active learning for training a defect classifier, comprising:
an imaging subsystem comprising at least an energy source and a detector, wherein the energy source is configured to generate energy that is directed to a specimen, and wherein the detector is configured to detect energy from the specimen and to generate images responsive to the detected energy; and one or more computer subsystems configured for performing active learning for training a defect classifier, wherein the active learning comprises:
applying an acquisition function to data points for the specimen, wherein the acquisition function selects one or more of the data points based on uncertainty estimations associated with the data points;
acquiring labels for the selected one or more data points; and
generating a set of labeled data comprising the selected one or more data points and the acquired labels; and
wherein the one or more computer subsystems are further configured for training the defect classifier using the set of labeled data, and wherein the defect classifier is configured for classifying defects detected on the specimen using the images generated by the imaging subsystem.

2 . The system of claim 1 , wherein the data points for the specimen consist of unlabeled data points.

3 . The system of claim 1 , wherein the data points for the specimen comprise a combination of fewer than ten ground truth data points for any one defect type and unlabeled data.

4 . The system of claim 1 , wherein the active learning further comprises repeating at least once a sequence of steps comprising the applying, the acquiring, and the generating steps, and wherein the generating step performed in the sequence of steps comprises appending the labels acquired for the one or more data points selected in the applying step performed in the sequence of steps to the set of labeled data.

5 . The system of claim 1 , wherein the acquisition function is defined as an unsupervised sampling method.

6 . The system of claim 1 , wherein the acquisition function is defined as a supervised sampling method.

7 . The system of claim 1 , wherein the acquisition function is defined as a semi-supervised sampling method.

8 . The system of claim 1 , wherein the acquisition function is defined as a combination of a supervised and unsupervised sampling method.

9 . The system of claim 1 , wherein the acquisition function is defined as a sampling method based on Maximum Entropy.

10 . The system of claim 1 , wherein the acquisition function is defined as a sampling method based on Bayesian Active Learning.

11 . The system of claim 1 , wherein the acquisition function is defined as an Error Reduction method.

12 . The system of claim 1 , wherein the acquisition function is defined as a Variance Reduction method.

13 . The system of claim 1 , wherein the acquisition function is defined as a deep learning model.

14 . The system of claim 1 , wherein the acquisition function is defined as a machine learning model.

15 . The system of claim 1 , wherein applying the acquisition function comprises estimating the acquisition function by evaluating one or more probability distributions using a Bayesian learning model.

16 . The system of claim 15 , wherein the Bayesian learning model is a Bayesian deep learning model.

17 . The system of claim 15 , wherein the Bayesian learning model is a Bayesian machine learning model.

18 . The system of claim 15 , wherein the one or more probability distributions comprise an unsupervised estimation of sample probability of one or more of the images generated by the imaging subsystem.

19 . The system of claim 15 , wherein the one or more probability distributions comprise a supervised or semi-supervised estimation of model posterior and its derived uncertainty distribution.

20 . The system of claim 1 , wherein acquiring the labels comprises classifying the selected one or more data points using a ground truth method.

21 . The system of claim 1 , wherein acquiring the labels comprises classifying the selected one or more data points through human input.

22 . The system of claim 1 , wherein acquiring the labels comprises classifying the selected one or more data points through a crowd sourcing method.

23 . The system of claim 1 , wherein acquiring the labels comprises classifying the selected one or more data points through physics simulation.

24 . The system of claim 1 , wherein the defect classifier is further configured as a nuisance event filter.

25 . The system of claim 1 , wherein the defect classifier is further configured as a defect detector.

26 . The system of claim 1 , wherein the defect classifier is further configured as an automatic defect classifier.

27 . The system of claim 1 , wherein the defect classifier is further configured as a multi-class classifier.

28 . The system of claim 1 , wherein the acquisition function is configured to select the one or more of the data points that have the highest probability of being new defect types.

29 . The system of claim 1 , wherein the acquisition function is configured to select the one or more of the data points that have the highest uncertainty of being any known defect type.

30 . The system of claim 1 , wherein the imaging subsystem is configured as an optical inspection subsystem.

31 . The system of claim 1 , wherein the imaging subsystem is configured as an electron beam inspection subsystem.

32 . The system of claim 1 , wherein the imaging subsystem is configured as an electron beam defect review subsystem.

33 . The system of claim 1 , further comprising an additional imaging subsystem configured to generate additional images for the specimen, wherein the one or more computer subsystems are further configured for performing hybrid inspection by detecting defects on the specimen using the images generated by the imaging subsystem and the additional images generated by the additional imaging subsystem.

34 . The system of claim 33 , wherein one of the imaging and additional imaging subsystems is configured as an optical imaging subsystem, and wherein the other of the imaging and additional imaging subsystems is configured as an electron beam imaging subsystem.

35 . The system of claim 1 , wherein the specimen comprises a wafer.

36 . The system of claim 1 , wherein the specimen comprises a reticle.

37 . A non-transitory computer-readable medium, storing program instructions executable on one or more computer systems for performing a computer-implemented method for performing active learning for training a defect classifier, wherein the computer-implemented method comprises:
performing active learning for training a defect classifier, wherein the active learning comprises:
applying an acquisition function to data points for a specimen, wherein the acquisition function selects one or more of the data points based on uncertainty estimations associated with the data points;
acquiring labels for the selected one or more data points; and
generating a set of labeled data comprising the selected one or more data points and the acquired labels; and
t

Assignees

• Kla Tencor Corp

Inventors

• Zhang Jing
• Dong Yujie
• Duffy Brian
• Wallingford Richard
• Daino Michael
• Bhaskar Kris

Classifications

• G06N20/00

  Machine learning · CPC title

• G06N3/08

  Learning methods · CPC title

• G06F18/40

  Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title