System and method for configuring a classifier to achieve a target error rate

Patent metadata

Field	Value
Publication number	US-2022019859-A1
Application number	US-202016931810-A
Country	US
Kind code	A1
Filing date	Jul 17, 2020
Priority date	Jul 17, 2020
Publication date	Jan 20, 2022
Grant date	—

Abstract

A computer-implemented method comprises training, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training; storing each threshold lookup table in memory; obtaining a target error rate; obtaining a new input and running the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and selecting a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
training, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training; storing each threshold lookup table in memory; obtaining a target error rate; obtaining a new input and running the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and selecting a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.

2 . The method of claim 1 , wherein the first classifier runs each input of the validation set of the input through the second classifier at different thresholds to determine when the second classifier issues the classification error.

3 . The method of claim 2 , wherein the different thresholds are incremented by a predefined value.

4 . The method of claim 1 , wherein the classification error is one of a false positive and a false negative.

5 . The method of claim 1 , wherein the number of data buckets is 2 k data buckets, where k is a number of nodes in a chokepoint layer in the first classifier.

6 . The method of claim 1 , wherein obtaining the target error rate comprises:
providing the user with a plurality of error rates for selection as the target error rate; and receiving a user selection of the target error rate.

7 . The method of claim 1 , wherein obtaining the target error rate comprises:
providing the user with a plurality of policy levels, each policy level associated with a particular error rate; and receiving a user selection of one of the policy levels.

8 . The method of claim 1 , wherein the target error rate is one of a false positive rate or a false negative rate.

9 . The method of claim 1 , further comprising:
setting the threshold for the second classifier; and running the new input through the second classifier to classify the new input.

10 . A system comprising:
at least one processor; and a memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, configure the at least one processor to:
train, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training;
store each threshold lookup table in memory;
obtain a target error rate;
obtain a new input and run the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and
select a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.

11 . The system of claim 10 , wherein the first classifier runs each input of the validation set of the input through the second classifier at different thresholds to determine when the second classifier issues the classification error.

12 . The system of claim 11 , wherein the different thresholds are incremented by a predefined value.

13 . The system of claim 10 , wherein the classification error is one of a false positive and a false negative.

14 . The system of claim 10 , wherein the number of data buckets is 2 k data buckets, where k is a number of nodes in a chokepoint layer in the first classifier.

15 . The system of claim 10 , wherein obtaining the target error rate comprises:
providing the user with a plurality of error rates for selection as the target error rate; and receiving a user selection of the target error rate.

16 . The system of claim 10 , wherein obtaining the target error rate comprises:
providing the user with a plurality of policy levels, each policy level associated with a particular error rate; and receiving a user selection of one of the policy levels.

17 . The system of claim 10 , wherein the target error rate is one of a false positive rate or a false negative rate.

18 . The system of claim 10 , wherein the instructions, when executed by the at least one processor, configure the at least one processor to:
set the threshold for the second classifier; and run the new input through the second classifier to classify the new input.

19 . A non-transitory computer readable storage medium comprising computer-executable instructions which, when executed, configure a processor to:
train, using a validation set of input, a first classifier to predict when a second classifier will issue a classification error on a particular input, the first classifier generating a number of data buckets based on the validation set of input and populating a threshold lookup table for each data bucket based on a number of thresholds set for the second classifier during the training; store each threshold lookup table in memory; obtain a target error rate; obtain a new input and run the new input through the first classifier, the first classifier selecting one of the data buckets for the new input; and select a threshold for the second classifier using the stored threshold lookup table for the selected data bucket and the target error rate.

20 . The non-transitory computer readable storage medium of claim 19 , wherein the target error rate is one of a false positive rate or a false negative rate.

Assignees

• Blackberry Ltd

Inventors

• Willis Edward Snow
• Henkel Steven John

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/2178 Primary

  based on feedback of a supervisor · CPC title

• G06F18/24

  Classification techniques · CPC title

• G06N3/048

  Activation functions · CPC title