Classifier readiness and maintenance in automatic defect classification

Patent metadata

Field	Value
Publication number	US-9607233-B2
Application number	US-201213452771-A
Country	US
Kind code	B2
Filing date	Apr 20, 2012
Priority date	Apr 20, 2012
Publication date	Mar 28, 2017
Grant date	Mar 28, 2017

Abstract

A method for classification includes receiving inspection data associated with a plurality of defects found in one or more samples and receiving one or more benchmark classification comprising a class for each of the plurality of defects. A readiness criterion for one or more of the classes is evaluated based on the one or more benchmark classification results, wherein the readiness criterion comprises for each class, a suitability of the inspection data for training an automatic defect classifier for the class. A portion of the inspection data is selected corresponding to one or more defects associated with one or more classes that satisfy the readiness criterion. One or more automatic classifiers are trained for the one or more classes that satisfy the readiness criterion using the selected portion of the inspection data.

First claim

The invention claimed is:

1. A method for classification, comprising:
receiving, by a computer system, inspection data associated with a plurality of defects found in one or more samples;
receiving, by the computer system, one or more benchmark classification results comprising a class for each of the plurality of defects;
evaluating, by the computer system, a readiness criterion for one or more of the classes based on the one or more benchmark classification results, wherein the readiness criterion for a given class is indicative of a suitability of the inspection data for training an automatic defect classifier for the given class, wherein the suitability of the inspection data for training the automatic defect classifier for the given class is indicated when the readiness criterion is satisfied, wherein the readiness criterion for the given class is satisfied when: an entire number of defects that have been classified in the benchmark classification results into borders of the given class is above a threshold number, and a ratio of a number of defects that have been classified into the borders of the given class with no overlapping with a classification associated with borders from another class to the entire number of defects that have been classified into the borders of the given class matches a predefined criterion;
selecting a portion of the inspection data corresponding to one or more defects associated with one or more classes that satisfy the readiness criterion; and
training one or more automatic classifiers for the one or more classes that satisfy the readiness criterion using the selected portion of the inspection data.

2. The method of claim 1 , wherein receiving the one or more benchmark classification results comprises receiving one or more classifications provided by a human inspector or other classification modality.

3. The method of claim 2 , further comprising:
applying the one or more trained automatic classifiers to further inspection data outside of the selected portion of the inspection data to generate a plurality of first classifications for the further inspection data;
obtaining a plurality of second classifications based on the further inspection data and on the plurality of first classifications; and
incorporating the further inspection data and the plurality of second classifications in the selected portion of the inspection data for use in further training of the one or more automatic classifiers.

4. The method of claim 1 , wherein evaluating the readiness criterion comprises accepting a class that can be classified with at least a threshold level of a performance measure using the inspection data, wherein the performance measure is at least one of accuracy, a classification performance measure, and a rejection performance measure.

5. The method of claim 1 , further comprising:
providing a result of the automatic classifiers to increase a consistency of manual classification; and
obtaining additional training data from the manual classification.

6. The method of claim 1 , further comprising:
providing a subset of the inspection data to an inspection modality for classification, wherein the subset of the inspection data comprises inspection data that could not be classified by the trained one or more automatic classifiers.

7. The method of claim 6 , wherein the inspection modality comprises a classification of defects in the subset of the inspection data by an operator.

8. The method of claim 6 , further comprising:
for a class that does not satisfy the readiness criterion,
identifying a group of defects associated with the class by the inspection modality;
collecting further inspection data from the subset of the inspection data used by the inspection modality in classifying defects in the group of defects;
adding the collected further inspection data to the selected portion of the inspection data to generate an augmented training set for the class;
determining whether the class satisfies the readiness criterion based on the augmented training set; and
upon determining that the class satisfied the readiness criterion based on the augmented training set, training an automatic classifier for the class using the augmented training set.

9. The method of claim 1 , further comprising:
upon detecting a change in a feature-space distribution of the plurality of defects in further inspection data outside the selected portion of the inspection data, retraining the one or more automatic classifiers or alerting an operator of a possible problem in production of the samples.

10. The method of claim 9 , wherein retraining the one or more automatic classifiers comprises selecting a subset of defects in the selected portion of the inspection data, wherein the selected subset of defects are selected to reflect a past distribution of the defects among the classes.

11. An apparatus comprising:
a memory; and
a processor to:
receive inspection data associated with a plurality of defects found in one or more samples and one or more benchmark classification results comprising a class for each of the plurality of defects;
evaluate a readiness criterion for one or more of the classes based on the one or more benchmark classification results, wherein the readiness criterion for a given class is indicative of a suitability of the inspection data for training an automatic defect classifier for the given class, wherein the suitability of the inspection data for training the automatic defect classifier for the given class is indicated when the readiness criterion is satisfied, wherein the readiness criterion for the given class is satisfied when: an entire number of defects that have been classified in the benchmark classification results into borders of the given class is above a threshold number, and a ratio of a number of defects that have been classified into the borders of the given class with no overlapping with a classification associated with borders from another class to the entire number of defects that have been classified into the borders of the given class matches a predefined criterion;
select a portion of the inspection data corresponding to one or more defects associated with one or more classes that satisfy the readiness criterion; and
train one or more automatic classifiers for the one or more classes that satisfy the readiness criterion using the selected portion of the inspection data.

12. The apparatus of claim 11 , wherein to receive the one or more benchmark classification results comprises receiving one or more classifications provided by a human inspector or other classification modality.

13. The apparatus of claim 12 , wherein the processor is further to apply the one or more trained automatic classifiers to further inspection data outside of the selected portion of the inspection data to generate a plurality of first classifications for the further inspection data, to obtain a plurality of second classifications based on the further inspection data and on the plurality of first classifications, and to incorporate the further inspection data and the plurality of second classifications in the selected portion of the inspection data for use in further training of the one or more automatic classifiers.

14. The apparatus of claim 11 , wherein to evaluate the readiness criterion, the processor is to accept a class that can be classified with at least a threshold level of a performance measure using the inspection data, wherein the performance measure is at least one of accuracy, a classification performance measure, and a rejection performance measure.

15. The apparatus

Assignees

• Kaizerman Idan
• Shlain Vladimir
• Rozenman Efrat
• Applied Materials Israel Ltd

Inventors

• Kaizerman Idan
• Shlain Vladimir
• Rozenman Efrat

Classifications

• G06T2207/30148

  Semiconductor; IC; Wafer · CPC title

• G06T7/0004

  Industrial image inspection · CPC title

• G06T7/001 Primary

  using an image reference approach · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/987

  with the intervention of an operator · CPC title