Method and apparatus for evaluating predictive model

Patent metadata

Field	Value
Publication number	US-10671933-B2
Application number	US-201715497658-A
Country	US
Kind code	B2
Filing date	Apr 26, 2017
Priority date	Jul 31, 2013
Publication date	Jun 2, 2020
Grant date	Jun 2, 2020

Abstract

An approach for a computer to evaluate a predictive model includes identifying features of training samples in a set of training samples. The approach selects evaluation metrics from a set of evaluation metrics as available metrics using identified features and includes determining recommended metrics using the predictive model, the available metrics, and a predetermined set of user-preferred metrics. The approach applies the predictive model created using the set of training samples to a set of test samples to calculate values of the available metrics. The approach evaluates the predictive model by using the available metrics and the values of the available metrics to evaluate the predictive model by evaluating the predictive model using the recommended metrics and the values of the recommended metrics.

First claim

The invention claimed is:

1. A method for evaluating a predictive model, comprising:
identifying, by one or more computer processors, features of training samples in a set of training samples;
selecting, by one or more computer processors, at least one evaluation metric from a set of evaluation metrics as one or more available metrics based on the identified features,
the selecting including determining one or more recommended metrics based on the predictive model, the one or more available metrics, and a predetermined first set of user-preferred metrics, wherein the predetermined first set of user-preferred metrics includes a plurality of second elements, each of which comprises a first user-preferred metric and at least one attribute associated with the first user-preferred metric, and wherein the at least one attribute associated with the first user-preferred metric at least comprises a weight indicating a degree of a user's preference to the first user-preferred metric;
applying, by one or more computer processors, the predictive model created based on the set of training samples to a set of test samples so as to calculate values of the one or more available metrics, wherein applying, by one or more computer processors, the predictive model created based on the set of training samples to a set of test samples so as to calculate values of the one or more available metrics comprises determining values of the one or more recommended metrics based on the values of the one or more available metrics; and
evaluating, by one or more computer processors, the predictive model by using the one or more available metrics and the values of the one or more available metrics, wherein evaluating, by one or more computer processors, the predictive model comprises evaluating the predictive model by using the one or more recommended metrics and the values of the one or more recommended metrics.

2. The method according to claim 1 , wherein the set of evaluation metrics comprises a plurality of first elements, each of which comprises an evaluation metric and at least one attribute associated with the evaluation metric.

3. The method according to claim 2 , wherein selecting, by one or more computer processors, at least one evaluation metric from the set of evaluation metrics as one or more available metrics based on the identified features comprises:
comparing, by one or more computer processors, the identified features with the at least one attribute associated with each evaluation metric in the set of evaluation metrics; and
in response to the identified features matching at least one attribute of at least one evaluation metric in the set of evaluation metrics, selecting, by one or more computer processors, the at least one evaluation metric as the one or more available metrics.

4. The method according to claim 2 , wherein the at least one attribute associated with the evaluation metric at least comprises a type of samples to which the evaluation metric is applicable, and a type of a data mining task to which the evaluation metric is applicable.

5. The method according to claim 1 , wherein the identified features at least comprise a type of the training samples and a type of a data mining task to which the training samples are directed.

6. The method according to claim 1 , wherein determining, by one or more computer processors, one or more recommended metrics comprises:
comparing, by one or more computer processors, the one or more available metrics with first user-preferred metrics in the predetermined first set of user-preferred metrics;
selecting, by one or more computer processors, one or more first user-preferred metrics matching the one or more available metrics from the first set of user-preferred metrics;
in response to a user's input indicating a desired number of the one or more recommended metrics, ranking, by one or more computer processors, the selected one or more first user-preferred metrics by weight; and
sequentially selecting, by one or more computer processors, the desired number of the one or more first user-preferred metrics as the one or more recommended metrics from the ranked one or more first user-preferred metrics.

7. The method according to claim 1 , wherein determining one or more recommended metrics further comprises:
receiving, by one or more computer processors, a second set of user-preferred metrics from a user;
comparing, by one or more computer processors, the one or more recommended metrics with second user-preferred metrics in the second set of user-preferred metrics; and
selecting, by one or more computer processors, one or more second user-preferred metrics matching the one or more recommended metrics from the second set of user-preferred metrics;
wherein determining values of the one or more recommended metrics comprises:

determining, by one or more computer processors, the values of the one or more recommended metrics matching the selected second user-preferred metrics as values of the selected second user-preferred metrics, and
wherein evaluating, by one or more computer processors, the predictive model comprises:

evaluating, by one or more computer processors, the predictive model by using the selected second user-preferred metrics and the values of the selected second user-preferred metrics.

8. The method according to claim 7 , wherein the second set of user-preferred metrics comprises a plurality of third elements, each of which at least comprises a second user-preferred metric and a weight associated with the second user-preferred metric.

9. The method according to claim 8 , further comprising:
updating, by one or more computer processors, the first set of user-preferred metrics by using the second user-preferred metrics and the weight of the second user-preferred metric.

10. A computer program product for evaluating a predictive model, the computer program product comprising:
one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions executable by a processor, the program instructions comprising instructions for:
identifying features of training samples in a set of training samples;
selecting at least one evaluation metric from a set of evaluation metrics as one or more available metrics based on the identified features,
the selecting including determining one or more recommended metrics based on the predictive model, the one or more available metrics, and a predetermined first set of user-preferred metrics, wherein the predetermined first set of user-preferred metrics comprises a plurality of second elements, each of which comprises a first user-preferred metric and at least one attribute associated with the first user-preferred metric, and wherein the at least one attribute associated with the first user-preferred metric at least comprises a weight indicating a degree of a user's preference to the first user-preferred metric;

applying the predictive model created based on the set of training samples to a set of test samples so as to calculate values of the one or more available metrics, wherein applying the predictive model created based on the set of training samples to a set of test samples so as to calculate values of the one or more available metrics comprises determining values of the one or more recommended metrics based on the values of the one or more available metrics; and
evaluating the predictive model by using the one or more available metrics and the values of the one or more available metrics, wherein evaluating the predictive model comprises evaluating the predictive model by using the one or more recommended metrics and the values of the one or more recom

Assignees

• IBM

Inventors

• Dong Wei S
• Tian Chunhua
• Wang Yu
• Wang Yu
• Yan Junchi
• Zhang Chao

Classifications

• G06Q10/06375

  Prediction of business process outcome or impact based on a proposed change · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title

• G06K9/6262

  Physics · mapped topic

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title

• G06N20/00 Primary

  Machine learning · CPC title