Performance checking component for an ETL job

Patent metadata

Field	Value
Publication number	US-9710530-B2
Application number	US-201615234139-A
Country	US
Kind code	B2
Filing date	Aug 11, 2016
Priority date	May 30, 2014
Publication date	Jul 18, 2017
Grant date	Jul 18, 2017

Abstract

Generation of a performance determination report for an Extract, Transform, Load (ETL) job includes decomposing the ETL job into two or more stage instances, and identifying one or more conditions for each of the stage instances. A set of tests for each of the identified conditions are generated. A first set of test results are generated by performing the set of tests. It is determined whether a test result from the first set of test results is outside of a first range. Conditions that can be identified include a non-volatile free memory condition, a network reliability condition, a network configuration condition, an application availability condition, a database availability condition, a database performance condition, a schema validity condition, an installed libraries condition, a configuration parameter condition, a volatile free memory condition, and a third party tool condition.

First claim

What is claimed is:

1. A computer-implemented method performed by a processor, comprising:
decomposing an Extract, Transform, and Load (ETL) job into two or more stage instances, the two or more stage instances including a first extraction stage instance and a second extraction stage instance, a first data validation stage instance and a second data validation stage instance, a transform stage instance, and a transfer stage instance;
identifying one or more conditions for each of the stage instances, the one or more conditions including a network reliability condition for the first and second extraction stage instances and for the transfer stage instance, the one or more conditions including a database performance condition for the first and second data validation stage instances, and the one or more conditions including a third party tool condition for the transform stage instance;
generating a set of tests for each of the one or more conditions, wherein each set of tests includes one or more tests for a corresponding condition, wherein the set of tests for the network reliability condition includes a ping test, the set of tests for the database performance condition includes a database query test, and the set of tests for the third party tool condition includes a test to validate transformed data values;
extracting data from a first source in the first extraction stage instance and extracting data from a second source in the second extraction stage instance;
validating data values for the first source in the first data validation stage instance and validating data values from the second source in the second data validation stage instance;
transforming the data from the first source and second source in the transform stage instance, generating a first joined data;
transferring the first joined data to one or more target databases in the transfer stage instance;
generating a first test result for the ETL job by performing a first set of tests during a runtime phase of the ETL job;
saving the first test result to an archive, the archive including data of a second test result and one or more resolutions for an underperforming condition, the second test result being generated by performing the first set of tests on a prior ETL job;
determining a first range for the first test result by calculating an average and a standard deviation from two or more historical test results, the two or more historical test results including the second test result, and wherein the first range is one standard deviation from the average;
determining that the first test result from the first set of test results is outside of the first range by comparing the first test result with the first range;
generating a performance determination report, the performance determination report including the first test result; and
providing the performance determination report to a user.

Assignees

• IBM

Inventors

• Li Jeff J
• Nusbickel Wendi L
• Tsimis James G

Classifications

• G06F11/3006

  where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50) · CPC title

• G06F11/3409

  for performance assessment · CPC title

• G06F17/30563 Primary

  Physics · mapped topic

• G06F11/2268

  Logging of test results · CPC title

• G06F11/302

  where the computing system component is a software system · CPC title