Data integration application execution management

Patent metadata

Field	Value
Publication number	US-2018288134-A1
Application number	US-201715478471-A
Country	US
Kind code	A1
Filing date	Apr 4, 2017
Priority date	Apr 4, 2017
Publication date	Oct 4, 2018
Grant date	—

Abstract

Disclosed aspects relate to data integration application execution management using both a set of managed nodes and a set of unmanaged nodes. An execution request for a data integration application may be sensed. A first set of commands of the data integration application may be assigned to the set of managed nodes. A second set of commands of the data integration application may be assigned to the set of unmanaged nodes. Execution of the data integration application may be initiated using the set of managed nodes to process the first set of commands and the set of unmanaged nodes to process the second set of commands.

First claim

What is claimed is:

1 . A computer-implemented method for data integration application execution management using both a set of managed nodes and a set of unmanaged nodes, the method comprising:
sensing an execution request for a data integration application; assigning, to execute the data integration application, a first set of commands of the data integration application to the set of managed nodes; assigning, to execute the data integration application, a second set of commands of the data integration application to the set of unmanaged nodes; and initiating execution of the data integration application using:
the set of managed nodes to process the first set of commands of the data integration application, and
the set of unmanaged nodes to process the second set of commands of the data integration application.

2 . The method of claim 1 , further comprising:
identifying that the set of unmanaged nodes do not belong to a managed cluster of nodes.

3 . The method of claim 1 , further comprising:
determining, based on a set of configuration data having a first set of indicators that corresponds to the first set of commands of the data integration application, to assign the first set of commands of the data integration application to the set of managed nodes; and determining, based on the set of configuration data having a second set of indicators that corresponds to the second set of commands of the data integration application, to assign the second set of commands of the data integration application to the set of unmanaged nodes.

4 . The method of claim 3 , further comprising:
structuring the first set of indicators to include a first set of tags coupled with a set of managed node identifiers for the set of managed nodes; and structuring the second set of indicators to include a second set of tags coupled with a set of unmanaged node identifiers for the set of unmanaged nodes.

5 . The method of claim 3 , further comprising:
configuring the first set of indicators to have one or more non-static node identification properties.

6 . The method of claim 3 , further comprising:
receiving, from a user, a user input related to the first and second sets of indicators; and generating, based on the user input, the set of configuration data having the first and second sets of indicators.

7 . The method of claim 1 , further comprising:
retrieving a set of managed node identifiers which corresponds to a managed cluster of nodes; and determining, using the set of managed node identifiers which corresponds to the managed cluster of nodes, to assign the first set of commands of the data integration application to the set of managed nodes and the second set of commands of the data integration application to the set of unmanaged nodes.

8 . The method of claim 7 , further comprising:
detecting, in a set of configuration data, a set of unspecified node identifiers; and comparing the set of unspecified node identifiers in the set of configuration data with the set of managed node identifiers which belong to the managed cluster of nodes.

9 . The method of claim 1 , further comprising:
communicating, to an external resource management tool, a request related to a set of computing resources for allocation; and receiving, by the set of managed nodes in response to the communication to the external resource management tool, the set of computing resources.

10 . The method of claim 9 , further comprising:
detecting a set of allocated node identifiers which indicates that the set of managed nodes has been allocated with the set of computing resources; and distributing the first set of commands of the data integration application to the set of managed nodes.

11 . The method of claim 9 , further comprising:
distributing the second set of commands of the data integration application to the set of unmanaged nodes, wherein the second set of commands utilizes a set of shell commands; and processing the second set of commands of the data integration application using the set of shell commands on the set of unmanaged nodes.

12 . The method of claim 1 , further comprising:
sensing an execution request for an Extract Transform and Load (ETL) job; assigning, to execute the ETL job, a first set of commands of the ETL job to the set of managed nodes; assigning, to execute the ETL job, a second set of commands of the ETL job to the set of unmanaged nodes; initiating execution of the ETL job using:
the set of managed nodes to process the first set of commands of the ETL job, and
the set of unmanaged nodes to process the second set of commands of the ETL job; and
processing, in parallel, the first and second sets of commands of the ETL job using both the set of managed nodes and the set of unmanaged nodes in a simultaneous fashion.

13 . The method of claim 1 , further comprising:
transferring, to perform the execution request of data integration application, a set of data between an unmanaged cluster of nodes and a managed cluster of nodes.

14 . The method of claim 1 , further comprising:
analyzing, to perform the execution request of data integration application, a set of data having both a first subset of the set of data which is stored on a managed cluster of nodes and a second subset of the set of data which is stored on an unmanaged cluster of nodes.

15 . The method of claim 14 , further comprising:
analyzing, on the unmanaged cluster of nodes to perform the execution request of the data integration application, the second subset of the set of data which is stored on the unmanaged cluster of nodes.

16 . The method of claim 1 , further comprising:
detecting that a managed cluster of nodes lacks a computing resource tool to process a set of data; detecting that an unmanaged cluster of nodes has the computing resource tool to process the set of data; and processing, to perform the execution request of data integration application, the set of data on the unmanaged cluster of nodes.

17 . The method of claim 1 , further comprising:
detecting that a managed cluster of nodes is in a cloud environment; detecting that an unmanaged cluster of nodes is in an on-premises environment; and processing, in parallel in a simultaneous fashion, the first and second sets of commands of the data integration application using both the set of managed nodes in the cloud environment and the set of unmanaged nodes in the on-premises environment.

18 . The method of claim 1 , further comprising:
performing, in a dynamic fashion to streamline data integration application execution management using both the set of managed nodes and the set of unmanaged nodes, each of:
the sensing, the assigning of the first set of commands, the assigning of the second set of commands, and the initiating execution; and
performing, in an automated fashion without user intervention, each of:
the sensing, the assigning of the first set of commands, the assigning of the second set of commands, and the initiating execution.

19 . A system for data integration application execution management using both a set of managed nodes and a set of unmanaged nodes, the system comprising:
a memory having a set of computer readable computer instructions, and a processor for executing the set of computer readable instructions, the set of computer readable instructions including: sensing an execution request for a data integration application; assigning, to execute the data integration application, a first set of commands

Assignees

• IBM

Inventors

• Bonagiri Krishna K
• Chintakayala Kalyanji
• Jacobson Eric A

Classifications

• H04L41/0803

  Configuration setting · CPC title

• G06F16/254

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• H04L67/10 Primary

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title

• H04L41/024

  using relational databases for representation of network management data, e.g. managing via structured query language [SQL] · CPC title

• H04L47/70

  Admission control; Resource allocation · CPC title