Supporting ETL processing in BPEL-based processes

Patent metadata

Field	Value
Publication number	US-9542656-B2
Application number	US-55940206-A
Country	US
Kind code	B2
Filing date	Nov 13, 2006
Priority date	Nov 13, 2006
Publication date	Jan 10, 2017
Grant date	Jan 10, 2017

Abstract

Methods and apparatus, including computer program products, implementing and using techniques for integrating and data activities in a process flow. A data transformation activity is invoked through local or remote invocation. The data transformation activity is part of a process flow defined in a standard business process execution language format and is invoked from within the process flow. A system for executing a process flow including one or more control activities and one or more data transformation activities is also described. The system includes a process control engine for executing activities included in the process flow, a data transformation subsystem for storing domain specific definitions of data transformation processes of data in one or more databases, and a control data repository for storing domain specific activity information related to the process flow.

First claim

The invention claimed is:

1. A computer program product for integrating control activities and extract, transform and load activities in a process flow for a data warehouse, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, wherein the program instructions are executable on a computer to cause the computer to perform a method comprising:
deploying a process definition file into a system comprising a process control engine for executing a process flow described in the process definition file, and one or more data transformation systems for performing the extract, transform and load activities, wherein the process definition file includes:
references to local and remote extract, transform and load activities, wherein each activity includes domain specific attribute values,
automatically generated process-scoped variables for each extract, transform and load activity, wherein the process-scoped variables hold and enable sharing of the state of each extract, transform and load activity,
references to specific dataflow execution metadata for dataflow activities included in the process flow,
a partner definition for each remote extract, transform and load service host, and
global process-scoped variables for holding pre-defined message definitions for the remote extract, transform and load services;

processing the process definition file, by the process control engine, to execute an activity described in the process definition file;
in executing the activity, determining, by the process control engine, whether the extract, transform and load activity is a local activity; and
in response to determining that the extract, transform and load activity is a local activity, invoking, by the process control engine, the extract, transform and load activity on data residing in a data warehouse through local invocation, by performing a method call to a service for local invocation in a data transformation processing subsystem.

2. The computer program product of claim 1 , wherein the computer readable program when executed on a computer further causes the computer to:
define one or more extract, transform and load activities as part of the process definition file.

3. The computer program product of claim 1 , wherein the computer readable program when executed on a computer further causes the computer to:
include a template partner link definition in the process definition file, the template partner link being operable to invoke a remote data transformation system.

4. The computer program product of claim 3 , wherein the template partner link definition is operable to be used as a placeholder partner link when the remote data transformation system has not been identified at design time of the process flow described in the process definition file.

5. The computer program product of claim 3 , wherein the template partner link definition is operable to be used as a map to predefined partner links when the remote data transformation system has been identified at design time of the process flow described in the process definition file.

6. The computer program product of claim 1 , wherein the computer readable program when executed on a computer further causes the computer to:
provide an abstraction layer as part of the process definition file.

7. The computer program product of claim 6 , wherein the abstraction layer includes a control flow level and a dataflow level.

8. The computer program product of claim 7 , wherein the dataflow level contains specifics about extract, transform and load activities.

9. The computer program product of claim 7 , wherein the control flow level contains specifics about dependencies and an execution sequence of the dataflow.

10. The computer program product of claim 7 , wherein the control flow level includes one or more transactional and error recovery options.

11. The computer program product of claim 1 , wherein the process control engine is a business process execution language engine.

12. The computer program product of claim 1 , wherein the computer readable program when executed on a computer further causes the computer to invoke a data movement that is a precursor to an extract, transform and load activity.

13. The computer program product of claim 1 , wherein the computer readable program when executed on a computer further causes the computer to:
invoke an extract, transform and load activity through remote invocation, in response to determining that the extract, transform and load activity is a remote activity.

14. The computer program product of claim 13 , wherein the invocation is done by calling a service for remote invocation in the data transformation processing subsystem.

15. The computer program product of claim 1 , wherein each dataflow activity and each control activity has a unique identifier.

16. A system for executing a process flow including one or more control activities and one or more extract, transform and load activities for a data warehouse, the system comprising:
a process control engine operable to execute activities included in the process flow, wherein the process flow is described in a process definition file, wherein the process definition file includes:
references to local and remote extract, transform and load activities, wherein each activity includes domain specific attribute values,
automatically generated process-scoped variables for each extract, transform and load activity, wherein the process-scoped variables hold and enable sharing of the state of each extract, transform and load activity,
references to specific dataflow execution metadata for dataflow activities included in the process flow,
a partner definition for each remote extract, transform and load service host, and
global process-scoped variables for holding pre-defined message definitions for the remote extract, transform and load services;

a data transformation subsystem operable to store domain specific definitions of extract, transform and load activities of data residing in one or more databases in a data warehouse, the data transformation subsystem further being operable to be invoked as one of: a local service and a remote service; and
a control data repository operable to store domain specific activity information related to the process flow.

17. The system of claim 16 , wherein the control data repository stores one or more of: runtime data for the process flow, auditing information for the process flow, and statistics generated from the execution of the activities in the process flow.

18. The system of claim 16 , wherein the process control engine further includes a business process execution language interface and a web services description language interface for communicating with other system components, wherein the business process execution language interface and the web services description language interface include the data from the process definition file.

19. The system of claim 16 , wherein the data transformation subsystem is operable to store domain specific information as a two-level abstraction layer on top of a standard business prosecution execution language definition.

20. The system of claim 19 , wherein the abstraction layer includes a control flow level and a dataflow level.

21. The system of claim 20 , wherein the dataflow level contains specifics about data transformations.

Assignees

• Behnen Marion
• Jin Qi
• Saillet Yannick
• Srinivasan Sriram
• Thirunavukkarasu Muthukumar
• Yoo Hoi J
• IBM

Inventors

• Behnen Marion
• Jin Qi
• Saillet Yannick
• Srinivasan Sriram
• Thirunavukkarasu Muthukumar
• Yoo Hoi J

Classifications

• G06Q10/103

  Workflow collaboration or project management · CPC title

• G06Q10/00 Primary

  Administration; Management · CPC title