Re-sizing data partitions for ensemble models in a mapreduce framework

Patent metadata

Field	Value
Publication number	US-9798782-B2
Application number	US-201414297606-A
Country	US
Kind code	B2
Filing date	Jun 5, 2014
Priority date	Jun 5, 2014
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

Techniques are described for revising data partition size for use in generating predictive models. In one example, a method includes determining an initial number of base model partitions of data from a plurality of data sources; determining an initial base model partition size based at least in part on the initial number of base model partitions; and evaluating the initial base model partition size at least in part with reference to at least one base model partition size reference. The method further includes determining a finalized number of base model partitions based at least in part on the initial base model partition size; determining a revised base model partition size; and generating revised base models based at least in part on the revised base model partition size, including using a predictive modeling framework to randomly assign input data records from the plurality of data sources into the base model partitions.

First claim

What is claimed is:

1. A computer program product comprising a non-transitory computer-readable medium having program code embodied therewith, the program code executable by a computing device to:
Determine a candidate adjustment factor for a number of base model partitions of data from a plurality of data sources based at least in part on a target size of an ensemble model;
Determine an initial number of the base model partitions as a sum of a target size of the ensemble model and the lower of either the candidate adjustment factor or a constant;
Determine an initial base model partition size based at least in part on the initial number of base model partitions;
Evaluate the initial base model partition size at least in part with reference to at least one base model partition size reference;
Determine a finalized number of base model partitions based at least in part on the evaluating of the initial base model partition size at least in part with reference to the at least one base model partition size reference;
Determine a revised base model partition size based at least in part on the finalized number of base model partitions; and
Generate revised base models based at least in part on the revised base model partition size, wherein generating the revised base models comprises using a predictive modeling framework to randomly assign input data records from the plurality of data sources into the finalized number of base model partitions.

2. A computer system comprising:
One or more processors, one or more computer-readable memories, and one or more non-transitory computer-readable storage devices;
Program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to determine a candidate adjustment factor for a number of base model partitions of data from a plurality of data sources based at least in part on a target size of an ensemble model;
Program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to determine an initial number of the base model partitions as a sum of a target size of the ensemble model and the lower of either the candidate adjustment factor or a constant;
Program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to determine an initial base model partition size based at least in part on the initial number of base model partitions;
Program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to evaluate the initial base model partition size at least in part with reference to at least one base model partition size reference;
Program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to determine a finalized number of base model partitions based at least in part on the evaluating of the initial base model partition size at least in part with reference to the at least one base model partition size reference;
Program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to determine a revised base model partition size based at least in part on the finalized number of base model partitions; and
Program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to generate revised base models based at least in part on the revised base model partition size, wherein generating the revised base models comprises using a predictive modeling framework to randomly assign input data records from the plurality of data sources into the finalized number of base model partitions.

3. The computer program product of claim 1 , wherein the predictive modeling framework comprises a MapReduce framework.

4. The computer program product of claim 1 , wherein using the predictive modeling framework to randomly assign the input data records from the plurality of data sources into the finalized number of base model partitions comprises using a MapReduce framework to randomly assign the input data records from the plurality of data sources into the finalized number of base model partitions.

5. The computer program product of claim 1 , wherein the program code is further executable by the computing device to determine the initial base model partition size by determining a floor of a total number of records in a dataset, minus a size of a holdout sample of a size of a validation sample V, then divided by a number of base model partitions.

6. The computer program product of claim 1 , wherein the initial number of base model partitions comprises an estimated target number of base model partitions, wherein the program code is further executable by the computing device to determine the initial base model partitions size based also at least in part on a minimum number of base model partitions.

7. The computer program product of claim 1 , wherein the program code is further executable by the computing device to generate an output based at least in part on the revised base model partitions size.

8. The computer program product of claim 1 , wherein the program code is further executable by the computing device to generate, using reduce operations, the plurality of base model partitions based on the plurality of training samples, prior to determining an initial number of base model partitions of data from a plurality of data sources.

9. The computer program product of claim 1 , wherein the program code is further executable by the computing device to generate the ensemble model based on the plurality of revised base models.

10. The computer program product of claim 9 , wherein the program code is further executable by the computing device to:
generate, using map operations for each of the data sources, a validation sample and a holdout sample from the data sources;
merge the holdout samples into a holdout dataset; and
generate, using a reduce operation, a reference model based on the validation samples.

11. The computer program product of claim 10 , wherein the program code is further executable by the computing device to evaluate the ensemble model and the reference model with reference to the holdout dataset.

12. The computer program product of claim 11 , wherein the program code is further executable by the computing device to generate a predictive model based at least in part on the evaluating of the ensemble model and the reference model with reference to the holdout dataset.

Assignees

• IBM

Inventors

• Dagli C Ates
• Mccarroll Niall Fraser
• Vasilenko Dmitry

Classifications

• G06F16/2465 Primary

  Query processing support for facilitating data mining operations in structured databases · CPC title

• G06F16/289

  Object oriented databases · CPC title

• G06F2216/03

  Data mining · CPC title

• G06F16/212

  with details for data modelling support · CPC title

• G06F17/30539 Primary

  Physics · mapped topic