Tuning of parameters for automatic classification

Patent metadata

Field	Value
Publication number	US-2016189055-A1
Application number	US-201414588101-A
Country	US
Kind code	A1
Filing date	Dec 31, 2014
Priority date	Dec 31, 2014
Publication date	Jun 30, 2016
Grant date	—

Abstract

A method, system and computer software product for tuning a classification system. The tuning method receives training data including items, each associated with a training class label, and obtains test data including association of each item with an automatic class label and corresponding values of a first confidence level and a second confidence level. Per automatic class, the method generates two or more performance metrics based on the training data and the test data. The method selects, for each automatic class, a preferred pair of values of the first confidence threshold and the second confidence threshold for which, by rejecting all items bellow the first and second thresholds, with respect to all of the automatic classes, a global optimum condition of the performance metrics is met.

First claim

1 . A method comprising:
receiving, by a processing device, training data including items, each associated with a training class label; obtaining test data including an association of each item with an automatic class label and corresponding values of a first confidence level and a second confidence level; per each automatic class, generating two or more performance metrics based on the training data and the test data; selecting, for each automatic class, a preferred pair of values of the first confidence threshold and the second confidence threshold for which, by rejecting all items below the first and second confidence thresholds, with respect to all of the automatic classes, a global optimum condition of the performance metrics is met.

2 . The method according to claim 1 , wherein the global optimum condition is met under one or more performance constraints applied to the performance metrics.

3 . The method according to claim 1 , wherein the selecting of the preferred pair of values of the first confidence threshold and the second confidence threshold comprises:
for each automatic class, generating a group of candidate pairs of values; and selecting from among the candidate pairs of values, a preferred pair of values for which, with respect to all of the automatic classes, the global optimum condition of the performance metrics is met.

4 . The method according to claim 3 , wherein the preferred pair of values is selected based on input received from a user regarding one or more desired performance levels.

5 . The method according to claim 4 , further comprising:
plotting a graph representing a set of candidate pairs of values and allowing the user to use the graph for selecting the preferred pair of values from the set of candidate pairs of values.

6 . The method according to claim 5 , wherein the graph is constructed by defining a grid of a first performance metric on an x axis and finding a global optimum condition of a second performance metric for a y axis for each point of the first performance metric.

7 . The method according to claim 3 , wherein one or more performance constraints are applied to the group of candidate pairs of values to generate a group of permitted pair values, and wherein the preferred pair of values is selected from the group of permitted pair of values.

8 . The method according to claim 1 , wherein the items are suspected defects inspected on a semiconductor substrate.

9 . The method according to claim 1 , wherein obtaining test data is carried out by applying the classification rules to at least a portion of the training data, with the first confidence threshold and the second confidence threshold set to given values.

10 . The method according to claim 1 wherein the generating two or more performance metrics is performed by comparing the training class label with the automatic class labels.

11 . The method according to claim 1 , wherein generating two or more performance metrics is carried out by applying the classification rules to the training data multiple times, with the first confidence threshold and/or the second confidence threshold set to a different value each time.

12 . The method according to claim 1 , wherein the performance metrics relate to one or more performance measures from one or more of:
a purity measure representing items which were classified as belonging to one of the automatic classes and having the same training class and test class; an accuracy measure representing all items which are classified correctly; a rejection rate of majority items representing the number of items that the classification system should have classified as belonging to one of the automatic classes but is unable to classify with confidence; an item of interest rate representing the number of items that are identified correctly as belonging to specific automatic class; a minority extraction representing the number of items that are identified correctly as not belonging to automatic classes; and a false alarm rate representing a number of items that should have been rejected and are classified as belonging to one of the automatic classes, out of the total number of rejected items.

13 . The method according to claim 2 , wherein the performance constraint is selected from at least one of:
a minimal purity; a minimal accuracy; a maximal rejection rate of majority items; a minimal item of interest rate; a minimal minority extraction; a maximal false alarm rate; and a minimal confidence threshold value.

14 . The method according to claim 1 , wherein the first confidence threshold and second confidence threshold are selected from at least one of:
an ‘Unknown’ confidence threshold representing a confidence level for which, an item that is classified by a single-class classifier as belonging to an automatic class with confidence level below the ‘Unknown’ confidence threshold will be rejected; a ‘Cannot decide’ confidence threshold representing a confidence level for which, an item that is classified by a multi-class classifier as belonging to an automatic class with confidence level below the ‘Cannot decide’ confidence threshold will be rejected; and an ‘Item of interest’ confidence threshold representing a confidence level for which, an item that is classified by a multi-class and single-class classifiers as belonging to a specific automatic class with confidence level below the ‘Item of interest’ confidence threshold will be rejected.

15 . An apparatus for tuning a classification system, the apparatus comprising:
a memory ; a processor operatively coupled with the memory to: receive training data including items, each associated with a training class label; obtain test data including association of each item with an automatic class label and corresponding values of a first confidence level and a second confidence level;
wherein the processor is further configured for:
per automatic class, generate two or more performance metrics based on the training data and the test data; and select, for each automatic class, a preferred pair of values of the first confidence threshold and the second confidence threshold for which, by rejecting all items bellow the first and second thresholds, with respect to all of the automatic classes, a global optimum condition of the performance metrics is met.

16 . The apparatus according to claim 14 , wherein the processor is further to receive one or more performance constraints and achieve the global optimum condition under the one or more performance constraints applied to the performance metrics.

17 . The apparatus according to claim 14 , wherein the processor is further to select a preferred pair of values of the first confidence threshold and the second confidence threshold by:
for each automatic class, generate a group of candidate pairs of values; and select from among the candidate pairs of values, a preferred pair of values for which, with respect to all of the automatic classes, a global optimum condition of the performance metrics is met.

18 . The apparatus according to claim 16 , wherein the processor is further to receive input from a user regarding one or more of desired performance levels, and for selecting the preferred pair of values based on said input received from a user.

19 . The apparatus according to claim 17 , wherein the processor is further to:
provide an output to the user a graph representing a set of candidate pairs of values; and enable the user to use the graph for inputting said one or mor

Assignees

• Applied Materials Israel Ltd

Inventors

• Zvitia Orly

Classifications

• H10P74/23

  characterised by multiple measurements, corrections, marking or sorting processes · CPC title

• G06N99/005 Primary

  Physics · mapped topic

• Y02P90/02

  Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS] · CPC title

• G05B19/41875 Primary

  characterised by quality surveillance of production · CPC title

• G06N20/00

  Machine learning · CPC title