Processing data in data migration

Patent metadata

Field	Value
Publication number	US-9460171-B2
Application number	US-201314074996-A
Country	US
Kind code	B2
Filing date	Nov 8, 2013
Priority date	Nov 8, 2013
Publication date	Oct 4, 2016
Grant date	Oct 4, 2016

Abstract

A computer-implemented method for processing information related to an extract-transform-load (ETL) data migration, including aggregating operational metadata and determining: a plurality of metrics, organized by business object, corresponding to the migration; a number of business object instances not successfully loaded; a first end-to-end execution time for at least one business object; relevant input metadata; load readiness status per business object; impact of a business object that is not load ready by analyzing business process hierarchies; business object load readiness by reference to incomplete development status or data defects; scope per test cycle based, at least in part, upon business object load readiness; and high-priority defects of business objects that stop testing based, at least in part, upon analysis of business process hierarchies.

First claim

What is claimed is:

1. A computer implemented method for processing information related to an extract-transform-load (ETL) data migration including a plurality of business objects, the method comprising:
aggregating, by a data flow aggregator portion of an algorithmic aggregator framework, operational metadata produced by information integration ETL functions;
aggregating, for each business object of the plurality of business objects and by a conversion object aggregator of the algorithmic aggregator framework, a set of metric(s) relating to data conversion performed in the scope of the ETL data migration;
determining, by an area comparator of the algorithmic aggregator framework, a difference between a number of business object instances successfully loaded and a number of business object instances which have been at least attempted to be loaded;
projecting a projected end-to-end execution time for the ETL data migration on a by business object by business object basis based on at least one of the following: project planned time and/or historical data related to previous ETL data migration jobs; and
creating a dashboard display data set that corresponds to a dashboard display, with the dashboard display including a human understandable visual indication of each of the following with respect to at least one business object of the plurality of business objects: the operational metadata produced by information integration ETL functions, the a set of metric(s) relating to data conversion performed in the scope of the ETL data migration, the difference between a number of business object instances successfully loaded and a number of business object instances which have been at least attempted to be loaded, and the projected end-to-end execution time for the ETL data migration.

2. The method of claim 1 , further comprising:
determining, by one or more processors, records and/or instances of business objects that cannot be moved between any two areas of interest involved in the ETL data migration process by determining: records on reject links in ETL processes; records selected for elimination from the data migration process in response to the records being determined to be duplicates of other records, and/or records filtered out of the data migration process for other reasons;
in response to determining the records and/or the instances of business objects that cannot be moved between the two areas of interest involved in the ETL data migration process, adjusting area record counts and/or business object instance counts; and
comparing the adjusted counts between the two areas of interest.

3. The method of claim 1 , further comprising:
determining a total number of instances of business objects entering a data conversion process across all sources by linking data objects to business processes; and
reducing the total number of instances of business objects entering the data conversion process across all sources based on duplicate records selected for elimination from the data migration process to yield a reduced number of instances; and
wherein the dashboard display further includes a human understandable visual indication of the reduced number of instances.

4. The method of claim 1 , further comprising:
determining business object instances affected by data defects by connecting fields to records to conversion objects to business objects via a business process hierarchy;
wherein the dashboard display further includes a human understandable visual indication of the business object instances affected by data defects.

5. The method of claim 1 , further comprising:
determining an average end-to-end processing time for each instance of a business object on a target system from the instances of business objects which have been loaded during test runs;
wherein the dashboard display further includes a human understandable visual indication of the average end-to-end processing time for each instance of a business object on a target system from the instances of business objects which have been loaded during test runs.

6. The method of claim 1 , further comprising:
determining data movement status by the business object, a business process area, or a deployment phase;
wherein the dashboard display further includes a human understandable visual indication of the data movement status.

7. A computer program product for processing information related to an extract-transform-load (ETL) data migration including a plurality of business objects, the computer program product comprising a non-transitory computer readable storage medium having stored thereon:
first program instructions programmed to aggregate, by a data flow aggregator portion of an algorithmic aggregator framework, operational metadata produced by information integration ETL functions;
second program instructions programmed to aggregate, for each business object of the plurality of business objects and by a conversion object aggregator of the algorithmic aggregator framework, a set of metric(s) relating to data conversion performed in the scope of the ETL data migration;
third program instructions programmed to determine, by an area comparator of the algorithmic aggregator framework, a difference between a number of business object instances successfully loaded and a number of business object instances which have been at least attempted to be loaded;
fourth program instructions programmed to project a projected end-to-end execution time for the ETL data migration on a by business object by business object basis based on at least one of the following: project planned time and/or historical data related to previous ETL data migration jobs; and
fifth program instructions programmed to create a dashboard display data set that corresponds to a dashboard display, with the dashboard display including a human understandable visual indication of each of the following with respect to at least one business object of the plurality of business objects: the operational metadata produced by information integration ETL functions, the a set of metric(s) relating to data conversion performed in the scope of the ETL data migration, the difference between a number of business object instances successfully loaded and a number of business object instances which have been at least attempted to be loaded, and the projected end-to-end execution time for the ETL data migration.

8. A computer system for processing information related to an extract-transform-load (ETL) data migration including a plurality of business objects, the computer system comprising:
a processor(s) set; and
a non-transitory computer readable storage medium;
wherein:
the processor set is structured, located, connected and/or programmed to run program instructions stored on the non-transitory computer readable storage medium; and
the program instructions include:
first program instructions programmed to aggregate, by a data flow aggregator portion of an algorithmic aggregator framework, operational metadata produced by information integration ETL functions,
second program instructions programmed to aggregate, for each business object of the plurality of business objects and by a conversion object aggregator of the algorithmic aggregator framework, a set of metric(s) relating to data conversion performed in the scope of the ETL data migration,
third program instructions programmed to determine, by an area comparator of the algorithmic aggregator framework, a difference between a number of business object instances successfully loaded and a number of business object instances which have been at least attempted to be loaded,
fourth program instructions programmed to project a projected end-to-end execution time for the ETL data migration on a by

Assignees

• IBM

Inventors

• Marrelli Carl M
• Narayanan Ram S
• Oberhofer Martin
• Rashidi Solmaz

Classifications

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F17/30563 Primary

  Physics · mapped topic

• G06F17/30569

  Physics · mapped topic