Dynamically loading graph-based computations

Patent metadata

Field	Value
Publication number	US-9753751-B2
Application number	US-201414520588-A
Country	US
Kind code	B2
Filing date	Oct 22, 2014
Priority date	Jun 15, 2010
Publication date	Sep 5, 2017
Grant date	Sep 5, 2017

Abstract

Processing data includes: receiving units of work that each include one or more work elements, and processing a first unit of work using a first compiled dataflow graph ( 160 ) loaded into a data processing system ( 100 ) in response to receiving the first unit of work. The processing includes: analysis to determine a characteristic of the first unit of work; identifying one or more compiled dataflow graphs from graphs stored in a data storage system ( 107 ) that include at least some that were compiled for processing a unit of work having the determined characteristic; loading one of the identified compiled dataflow graphs into the data processing system ( 100 ) as the first compiled dataflow graph ( 160 ); and generating one or more output work elements from at least one work element in the first unit of work.

First claim

What is claimed is:

1. A method for processing data in a data processing system using compiled dataflow graphs stored in a data storage system, the method including:
receiving multiple units of work that each include one or more work elements; and
processing a unit of work of the multiple units of work, the processing including:
analyzing the unit of work to determine a characteristic of the unit of work;
identifying a compiled dataflow graph, from multiple compiled dataflow graphs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
loading the identified compiled dataflow graph into a data processing program being executed by the data processing system, wherein the data processing program is being executed before receipt of the unit of work; and
generating one or more output work elements from at least one work element in the unit of work using the identified compiled dataflow graph.

2. The method of claim 1 , wherein the loading further includes retrieving data structures and loading the retrieved data structures into the data processing program.

3. The method of claim 2 , wherein the identified compiled dataflow graph is based on a first dataflow graph and loading the retrieved data structures includes loading the retrieved data structures into a second dataflow graph by embedding the first dataflow graph within a containing component of the second dataflow graph that is configured to provide a first process to execute data processing components in the first dataflow graph.

4. The method of claim 3 , wherein the containing component of the second dataflow graph is further configured to provide a first application programming interface that the first process uses to communicate with a second process that accesses a library for a first remote database.

5. The method of claim 2 , wherein the identified compiled dataflow graph is based on a first dataflow graph and loading the retrieved data structures includes loading the retrieved data structures into a second dataflow graph by embedding the first dataflow graph within a containing component of the second dataflow graph, the embedding including:
connecting a flow of input work elements received from a link connected to an input of the containing component to an input of the first dataflow graph, and
connecting the generated one or more output work elements to a link connected to an output of the containing component.

6. The method of claim 2 , further including, after generating the one or more output work elements, unloading the retrieved data structures.

7. The method of claim 1 , further including loading another compiled dataflow graph into the data processing system, the other dataflow graph having been compiled into data structures.

8. The method of claim 1 , wherein analyzing the unit of work includes reading an identifier included within a work element of the unit of work.

9. The method of claim 1 , wherein analyzing the unit of work includes identifying one of multiple predetermined types characterizing a work element of the unit of work.

10. The method of claim 1 , wherein analyzing the unit of work includes identifying a property of content included within a work element of the unit of work.

11. The method of claim 1 , wherein analyzing the unit of work includes determining one or more values associated with the unit of work that are designated as values for a set of one or more parameters.

12. A data processing system for processing data using compiled dataflow graphs stored in a data storage system, the system including:
at least one processor; and
one or more machine-readable hardware storage devices storing instructions that are executable by the at least one processor to perform operations including:
receiving multiple units of work that each include one or more work elements; and
processing a unit of work of the multiple units of work, the processing including:
analyzing the unit of work to determine a characteristic of the unit of work;
identifying a compiled dataflow graph from multiple compiled dataflow graphs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
loading the identified compiled dataflow graph into a data processing program being executed by the data processing system, wherein the data processing program is being executed before receipt of the unit of work; and
generating one or more output work elements from at least one work element in the unit of work using the identified compiled dataflow graph.

13. The data processing apparatus of claim 12 , wherein the loading further includes retrieving data structures and loading the retrieved data structures into the data processing program.

14. The data processing apparatus of claim 13 , wherein the identified compiled dataflow graph is based on a first dataflow graph and loading the retrieved data structures includes loading the retrieved data structures into a second dataflow graph by embedding the first dataflow graph within a containing component of the second dataflow graph that is configured to provide a first process to execute data processing components in the first dataflow graph.

15. The data processing apparatus of claim 14 , wherein the containing component of the second dataflow graph is further configured to provide a first application programming interface that the first process uses to communicate with a second process that accesses a library for a first remote database.

16. The data processing apparatus of claim 13 , wherein the identified compiled dataflow graph is based on a first dataflow graph and loading the retrieved data structures includes loading the retrieved data structures into a second dataflow graph by embedding the first dataflow graph within a containing component of the second dataflow graph, the embedding including:
connecting a flow of input work elements received from a link connected to an input of the containing component to an input of the first dataflow graph, and
connecting the generated one or more output work elements to a link connected to an output of the containing component.

17. The data processing apparatus of claim 13 , wherein the operations further include after generating the one or more output work elements, unloading the retrieved data structures.

18. The data processing apparatus of claim 12 , wherein the operations further include loading another compiled dataflow graph into the data processing system, the other compiled dataflow graph having been compiled into data structures.

19. The data processing apparatus of claim 12 , wherein analyzing the unit of work includes reading an identifier included within a work element of the unit of work.

20. The data processing apparatus of claim 12 , wherein analyzing the unit of work includes identifying one of multiple predetermined types characterizing a work element of the unit of work.

21. The data processing apparatus of claim 12 , wherein analyzing the unit of work includes identifying a property of content included within a work element of the unit of work.

22. The data processing apparatus of claim 12 , wherein analyzing the unit of work includes determining one or more values associated with the unit of work that are designated as values for a set of one or more parameters.

23. A data processing system for processing data using

Assignees

• Ab Initio Technology Llc

Inventors

• Atterbury Matthew Darcy
• Bromley H Mark
• Mesard Wayne
• Popov Arkadi
• Schmidt Stephen
• Stanfill Craig W
• Wholey Joseph Skeffington

Classifications

• G06F9/44536 Primary

  Selecting among different versions · CPC title

• G06F9/44521 Primary

  Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading · CPC title

• G06F9/4436

  Physics · mapped topic

• G06F9/4494

  data driven · CPC title