Switching leader-endorser for classifier decision combination

Patent metadata

Field	Value
Publication number	US-10546247-B2
Application number	US-201514955547-A
Country	US
Kind code	B2
Filing date	Dec 1, 2015
Priority date	Dec 1, 2015
Publication date	Jan 28, 2020
Grant date	Jan 28, 2020

Abstract

An approach is provided in which an information handling system trains multiple classifiers using a set of training samples. The information handling system selects a leader classifier from the multiple classifiers that generates the most amount of correct decisions corresponding to the set of training samples. Next, the information handling system identifies an endorser classifier from the multiple classifiers that generates the highest proportion of correct decisions among the endorser classifier's decisions matching the leader classifier's decisions, and combines the leader classifier and the endorser classifier into a combined classifier stage. In turn, the information handling system utilizes the combined classifier stage to process inquiries and generate results.

First claim

The invention claimed is:

1. A method implemented by an information handling system that includes a memory and a processor, the method comprising:
training a plurality of classifiers based upon a set of training samples;
in response to the training, identifying a leader classifier that generates a first set of decisions comprising a most amount of correct decisions out of the plurality of classifiers;
for each of a selected one of the plurality of classifiers less the leader classifier:
generating a second set of decisions by the selected classifier during the training;
determining, from the second set of decisions, an amount of matching decisions that match the first set of decisions generated by the leader classifier;
identifying, from the matching decisions, an amount of matching correct decisions; and
computing an accuracy of matching decisions value by dividing the amount of matching correct decisions by the amount of matching decisions;

selecting, from the plurality of classifiers, an endorser classifier that has the highest accuracy of matching decisions value;
storing the leader classifier and the endorser classifier as a combined classifier stage into a pipeline;
removing a portion of the training samples from the set of training samples that correspond to the matching decisions, resulting in a subset of training samples;
retraining all of the plurality of classifiers on the subset of training samples in response to removing the portion of the training samples from the set of training samples corresponding to the matching decisions;
in response to retraining each of the plurality of classifiers on the subset of training samples, selecting a subsequent leader classifier and a subsequent endorser classifier as a subsequent combined classifier stage;
storing the subsequent combined classifier stage into the pipeline at a stage subsequent to the combined classifier stage; and
using the pipeline to generate one or more answers to a natural language question in a question answer system.

2. The method of claim 1 further comprising:
computing a leader classifier accuracy value by dividing the most amount of correct decisions in the first set of decisions by an amount of training samples in the set of training samples; and
combining the leader classifier with the endorser classifier into the combined classifier stage in response to determining that the endorser classifier's accuracy of matching decisions value is higher than the leader classifier accuracy value.

3. The method of claim 1 further comprising:
wherein the subsequent leader classifier generates a set of third decisions having a most amount of subsequent correct decisions; and
selecting the subsequent endorser classifier that generates a set of fourth decisions that match one or more of the third decisions, resulting in a set of subsequent matching decisions, wherein the selection of the endorser classifier is based upon comparing a total amount of the subsequent matching decisions against an amount of the subsequent matching decisions that are correct.

4. The method of claim 3 further comprising:
processing the inquiry using the combined classifier stage;
in response to the leader classifier and the endorser classifier generating one or more same decisions to the inquiry:
exiting the pipeline; and
providing the one or more same decisions as the one or more results to the inquiry; and

in response to the leader classifier and the endorser classifier not generating at least one same decision to the inquiry, processing the inquiry by the subsequent combined classifier stage.

5. The method of claim 3 further comprising:
in response to determining that the subsequent accuracy of matching decisions value is not greater than the subsequent leader classifier accuracy value, storing the subsequent leader classifier without the subsequent endorser classifier into the pipeline.

6. The method of claim 1 further comprising:
combining the leader classifier with the endorser classifier into the combined classifier stage in response to determining that the amount of matching decisions meets a required minimum amount of matching decisions.

7. The method of claim 1 further comprising:
adding a different endorser classifier with the endorser classifier and the leader classifier to the combined classifier stage, wherein at least one of the one or more answers are based upon the leader classifier, the endorser classifier, and the different endorser classifier agreeing on the at least one answer.

8. The method of claim 1 wherein the information handling system operates in a regression context, the method further comprising:
determining the amount of matching decisions based on one or more numerical response differences between the leader classifier and the endorser classifier, wherein the selection of the endorser classifier is based on a decision reliability value of the matching decisions; and
wherein the combined classifier stage generates one or more numerical responses to the inquiry in the regression context.

9. An information handling system comprising:
one or more processors;
a memory coupled to at least one of the processors; and
a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of:
training a plurality of classifiers based upon a set of training samples;
in response to the training, identifying a leader classifier that generates a first set of decisions comprising a most amount of correct decisions out of the plurality of classifiers;
for each of a selected one of the plurality of classifiers less the leader classifier:
generating a second set of decisions by the selected classifier during the training;
determining, from the second set of decisions, an amount of matching decisions that match the first set of decisions generated by the leader classifier;
identifying, from the matching decisions, an amount of matching correct decisions; and
computing an accuracy of matching decisions value by dividing the amount of matching correct decisions by the amount of matching decisions;

selecting, from the plurality of classifiers, an endorser classifier that has the highest accuracy of matching decisions value;
storing the leader classifier and the endorser classifier as a combined classifier stage into a pipeline;
removing a portion of the training samples from the set of training samples that correspond to the matching decisions, resulting in a subset of training samples;
retraining all of the plurality of classifiers on the subset of training samples in response to removing the portion of the training samples from the set of training samples corresponding to the matching decisions;
in response to retraining each of the plurality of classifiers on the subset of training samples, selecting a subsequent leader classifier and a subsequent endorser classifier as a subsequent combined classifier stage;
storing the subsequent combined classifier stage into the pipeline at a stage subsequent to the combined classifier stage; and
using the pipeline to generate one or more answers to a natural language question in a question answer system.

10. The information handling system of claim 9 wherein at least one of the one or more processors perform additional actions comprising:
computing a leader classifier accuracy value by dividing the most amount of correct decisions in the first set of decisions by an amount of training samples in the set of training samples; and
combining the leader classifier with the endorser classifier into the combined classifier stage in response to determining that the endorser classifier's accuracy

Assignees

• IBM

Inventors

• Ho Tin Kam
• Lastras-Montano Luis A
• Misra Vinith

Classifications

• G06N5/04

  Inference or reasoning models · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06N20/00 Primary

  Machine learning · CPC title