Technology for extensible in-memory computing

Patent metadata

Field	Value
Publication number	US-10380137-B2
Application number	US-201615290544-A
Country	US
Kind code	B2
Filing date	Oct 11, 2016
Priority date	Oct 11, 2016
Publication date	Aug 13, 2019
Grant date	Aug 13, 2019

Abstract

A user-defined function (UDF) is received in a central Computer System, which causes registration of the UDF and distributes the UDF to a cluster of computer system nodes configured for performing, in volatile memory of the nodes, extract-transform-load processing of data cached in the volatile memory of the nodes. First and second job specifications that include the UDF are received by the central Computer System, and the central computer system distributes instructions for the job specifications to the nodes including at least one instruction that invokes the UDF for loading and executing the UDF in the volatile memory of at least one of the nodes during runtime of the jobs. The central Computer System does not cause registration of the UDF again after receiving the first job specification.

First claim

What is claimed is:

1. A method comprising:
receiving a user-defined function (UDF) in a central Computer System, and the central computer system causing registration of the UDF and distributing the UDF to a cluster of computer system nodes configured for performing, in volatile memory of the nodes, extract-transform-load processing of data cached in the volatile memory of the nodes; and
receiving by the central Computer System first and second job specifications that specify the UDF, and the central computer system distributing instructions for the job specifications to the nodes, including at least one instruction that invokes the UDF for loading and executing the UDF in the volatile memory of at least one of the nodes during runtime of the job, where the central Computer System does not cause registration of the UDF again after receiving the first job specification.

2. The method of claim 1 , where the user-defined function provides a mapping function and the method includes executing the user-defined, mapping function in volatile memory of the nodes during extract-transform-load processing of data cached in the volatile memory of the nodes.

3. The method of claim 2 , where executing the user-defined, mapping function performs at least one of the following actions during extract-transform-load processing of data cached in volatile memory of the nodes:
accessing a database;
calculating data quality of data from the database; and
writing the data quality.

4. The method of claim 3 , comprising:
sharing resources among at least two software applications cached in nodes of the cluster while executing the user-defined, mapping function that performs at least one of the actions during extract-transform-load processing of data cached in volatile memory of the nodes.

5. The method of claim 1 , where the at least one instruction that invokes the user-defined function includes a Spark SQL instruction.

6. The method of claim 1 , comprising:
compiling the received job specifications to generate the instructions distributed to the nodes, where the first job specification having the at least one instruction that invokes the user-defined function is added to an earlier received job specification having no instruction that invokes the user-defined function, and where the earlier received job specification is compiled before receiving the first job specification and is not recompiled after adding the first job specification.

7. The method of claim 1 , comprising:
applying a pin operation to cause the nodes to cache at least one class file for the user-defined function in volatile memory of the nodes for a next job run.

8. A system comprising:
a processor; and
a computer readable storage medium connected to the processor, where the computer readable storage medium has stored thereon a program for controlling the processor, and where the processor is operative with the program to execute the program for:
receiving a user-defined function (UDF) in a central Computer System, and the central computer system causing registration of the UDF and distributing the UDF to a cluster of computer system nodes configured for performing, in volatile memory of the nodes, extract-transform-load processing of data cached in the volatile memory of the nodes; and
receiving by the central Computer System first and second job specifications that specify the UDF, and the central computer system distributing instructions for the job specifications to the nodes, including at least one instruction that invokes the UDF for loading and executing the UDF in the volatile memory of at least one of the nodes during runtime of the job, where the central Computer System does not cause registration of the UDF again after receiving the first job specification.

9. The system of claim 8 , where the user-defined function provides a mapping function and the where the processor is operative with the program to execute the program for:
executing the user-defined, mapping function in volatile memory of the nodes during extract-transform-load processing of data cached in the volatile memory of the nodes.

10. The system of claim 9 , where executing the user-defined, mapping function performs at least one of the following actions during extract-transform-load processing of data cached in volatile memory of the nodes:
accessing a database;
calculating data quality of data from the database; and
writing the data quality.

11. The system of claim 10 , where the processor is operative with the program to execute the program for:
sharing resources among at least two software applications cached in nodes of the cluster while executing the user-defined, mapping function that performs at least one of the actions during extract-transform-load processing of data cached in volatile memory of the nodes.

12. The system of claim 8 , where the at least one instruction that invokes the user-defined function includes a Spark SQL instruction.

13. The system of claim 8 , where the processor is operative with the program to execute the program for:
compiling the received job specifications to generate the instructions distributed to the nodes, where the first job specification having the at least one instruction that invokes the user-defined function is added to an earlier received job specification having no instruction that invokes the user-defined function, and where the earlier received job specification is compiled before receiving the first job specification and is not recompiled after adding the first job specification.

14. The system of claim 8 , where the processor is operative with the program to execute the program for:
applying a pin operation to cause the nodes to cache at least one class file for the user-defined function in volatile memory of the nodes for a next job run.

15. A computer program product, including a computer readable storage medium having instructions stored thereon for execution by a computer system, where the instructions, when executed by the computer system, cause the computer system to implement a method comprising:
receiving a user-defined function (UDF) in a central Computer System, and the central computer system causing registration of the UDF and distributing the UDF to a cluster of computer system nodes configured for performing, in volatile memory of the nodes, extract-transform-load processing of data cached in the volatile memory of the nodes; and
receiving by the central Computer System first and second job specifications that specify the UDF, and the central computer system distributing instructions for the job specifications to the nodes, including at least one instruction that invokes the UDF for loading and executing the UDF in the volatile memory of at least one of the nodes during runtime of the job, where the central Computer System does not cause registration of the UDF again after receiving the first job specification.

16. The computer program product of claim 15 , where the user-defined function provides a mapping function and the where the instructions, when executed by the computer system, cause the computer system to implement a method comprising:
executing the user-defined, mapping function in volatile memory of the nodes during extract-transform-load processing of data cached in the volatile memory of the nodes.

17. The computer program product of claim 16 , where executing the user-defined, mapping function performs at least one of the following actions during extract-transform-load processing of data cached in volatile memory of the nodes:
accessing a database;
calcul

Assignees

• IBM

Inventors

• Li Yong
• Pham Ryan
• Pu Xiaoyan

Classifications

• G06F16/24552

  Database cache management · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title