Dynamic optimization of workload execution based on statistical data collection and updated job profiling

Patent metadata

Field	Value
Publication number	US-9996389-B2
Application number	US-201414205080-A
Country	US
Kind code	B2
Filing date	Mar 11, 2014
Priority date	Mar 11, 2014
Publication date	Jun 12, 2018
Grant date	Jun 12, 2018

Abstract

Embodiments presented herein provide techniques for optimizing parallel data flows of a batch processing job using a profile of the processing job. An application retrieves a job profile for a processing job. The processing job has a plurality of processing stages specified in an execution profile. The job profile includes statistical data for at least one of the processing stages obtained during prior executions of the job. The application modifies properties of the execution profile based on the job profile to optimize the execution of the job. The application executes the processing job with the modified execution profile.

First claim

What is claimed is:

1. A system, comprising:
a processor; and
a memory storing a program, which, when executed on the processor, performs an operation, the operation comprising:
retrieving, via a processor, a job profile for a processing job, wherein the processing job has a plurality of processing stages specified in an execution profile, wherein the execution profile specifies a respective set of one or more parameters for each of one or more of the processing stages of the processing job, and wherein the job profile includes statistical data for at least one of the processing stages obtained during prior executions of the processing job, wherein the statistical data includes an amount of data that was spilled to a disk during at least one of the one or more processing stages, and wherein the statistical data was obtained by, for each of the processing stages during at least a first of the prior executions:
identifying a plurality of operations in the processing stage, and
during performance of each of the operations:
determining whether a flag is set for the operation, wherein the flag indicates to gather statistical data relating to the operation, and
upon determining that a flag is set, collecting the statistical data relating to the operation,

identifying a plurality of optimizations to apply to the execution profile based on the statistical data and based on an indication that at least a first one of the processing stages of the processing job was selected for optimization during one of the prior executions,
receiving, via a user interface, a selection of at least a first optimization of the identified optimizations to apply to the execution profile, wherein the first optimization comprises a selection of a compression algorithm;
modifying the respective set of one or more parameters of the execution profile for the at least the first one of the processing stages of the processing job by applying the selected first optimization to the at least the first one of the processing stages to optimize an execution environment used to execute the processing job, and
executing the processing job in the optimized execution environment based on the modified execution profile.

2. The system of claim 1 , wherein the statistical data includes at least one of a total data size processed in the processing stages, an average record size processed by data transfer blocks, and an amount of system resource usage for each processing stage.

3. The system of claim 1 , wherein the job profile was generated during a prior execution of the processing job by monitoring the processing stages during the prior execution.

4. The system of claim 1 , wherein the job profile comprises a plurality of job profiles generated during a corresponding plurality of prior executions of the processing job.

5. The system of claim 1 , wherein the processing job is an extract, transform, and load job.

6. The system of claim 1 , wherein the modified respective set of one or more parameters includes at least one of a transfer block size, table chaining, and a record layout specified in the execution profile.

7. A computer program product, comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to perform an operation, the operation comprising:
retrieving, via a processor, a job profile for a processing job, wherein the processing job has a plurality of processing stages specified in an execution profile, wherein the execution profile specifies a respective set of one or more parameters for each of one or more of the processing stages of the processing job, and wherein the job profile includes statistical data for at least one of the processing stages obtained during a prior execution of the processing job, wherein the statistical data includes an amount of data that was spilled to a disk during at least one of the one or processing stages, and wherein the statistical data was obtained by, for each of the processing stages during at least a first of the prior executions:
identifying a plurality of operations in the processing stage, and
during performance of each of the operations:
determining whether a flag is set for the operation, wherein the flag indicates to gather statistical data relating to the operation, and
upon determining that a flag is set, collecting the statistical data relating to the operation;

identifying a plurality of optimizations to apply to the execution profile based on the statistical data and based on an indication that at least a first one of the processing stages of the processing job was selected for optimization during one of the prior executions,
receiving, via a user interface, a selection of at least a first optimization of the identified optimizations to apply to the execution profile, wherein the first optimization comprises a selection of a compression algorithm;
modifying the respective set of one or more parameters of the execution profile for the at least the first one of the processing stages of the processing job by applying the selected first optimization to the at least the first one of the processing stages to optimize an execution environment used to execute the processing job; and
executing the processing job in the optimized execution environment based on the modified execution profile.

8. The computer program product of claim 7 , wherein the statistical data includes at least one of a total data size processed in the processing stages, an average record size processed by data transfer blocks, and an amount of system resource usage for each processing stage.

9. The computer program product of claim 7 , wherein the job profile was generated during a prior execution of the processing job by monitoring the processing stages during the prior execution.

10. The computer program product of claim 7 , wherein the operation further comprises, receiving a specification of the parameters of the execution profile which are available to modify based on the job profile.

11. The computer program product of claim 7 , wherein the job profile comprises a plurality of job profiles generated during a corresponding plurality of prior executions of the processing job.

12. The computer program product of claim 7 , wherein the processing job is an extract, transform, and load job.

13. The computer program product of claim 7 , wherein the modified respective set of one or more parameters includes at least one of a transfer block size, table chaining, and a record layout specified in the execution profile.

Assignees

• IBM

Inventors

• Caufield Brian K
• Greene Lawrence A
• Jacobson Eric A
• Li Yong
• Pu Xiaoyan

Classifications

• G06F11/34

  Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation {; Recording or statistical evaluation of user activity, e.g. usability assessment} · CPC title

• G06F9/4843 Primary

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/50

  Allocation of resources, e.g. of the central processing unit [CPU] · CPC title