Managing interfaces for sub-graphs

Patent metadata

Field	Value
Publication number	US-10180821-B2
Application number	US-201414561435-A
Country	US
Kind code	B2
Filing date	Dec 5, 2014
Priority date	Dec 5, 2013
Publication date	Jan 15, 2019
Grant date	Jan 15, 2019

Abstract

Combining specifications of dataflow graphs includes receiving: a first dataflow graph specification that specifies two or more components connected by links representing flows of data, and a second dataflow graph specification that specifies at least one component, and at least one sub-graph interface. The sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. The method includes processing information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph specification, including: identifying an association between the sub-graph interface and the first dataflow graph specification, for at least a first flow junction, determining a direction associated with transferring a value of a descriptor, and transferring a value of a descriptor according to the determined direction.

First claim

What is claimed is:

1. A method for combining specifications of dataflow graphs, the method including:
receiving over an input device or port of a computer that includes at least one processor device, a plurality of dataflow graph specifications, including:
a first dataflow graph specification that specifies two or more components including a first component and a second component, which components are connected by links representing flows of data between components, and the components representing computations applied to flows of data into ports of the components or out of ports of the components, and
a second dataflow graph specification that specifies at least a third component, and at least one sub-graph interface that includes a flow junction representing a connection between a flow of data outside the sub-graph interface to or from a port of the third component, and a flow of data inside the sub-graph interface to or from a port of a component of the first dataflow graph specification; and

processing, by the at least one processor device, information including the first dataflow graph specification and the second dataflow graph specification to generate a combined dataflow graph specification, with the processing including:
identifying an association between the sub-graph interface and the first dataflow graph specification,
determining for the flow junction on the sub-graph interface, a direction associated with a transfer of a value of a descriptor of a data or a computational characteristic, where the determined direction is one of an inward transfer of the value of the descriptor from the flow junction on the sub-graph interface to a component of the first dataflow graph specification or an outward transfer of the value of the descriptor from the flow junction on the sub-graph interface to a component of the second dataflow graph specification, and

causing, by the at least one processor device, a transfer of the value according to the determined direction.

2. The method of claim 1 , wherein the first dataflow graph specification includes at least one indicator that a descriptor associated with the first component is identical to a descriptor associated with the second component.

3. The method of claim 2 , wherein the determined direction corresponds to the inward transfer of the value of the descriptor from the flow junction on the sub-graph interface to the second component.

4. The method of claim 3 , wherein the descriptor is provided to the flow junction from the third component.

5. The method of claim 3 , wherein the descriptor is provided to the first component from the second component.

6. The method of claim 1 , wherein the second dataflow graph specification includes at least one indicator that a descriptor associated with the third component is identical to a descriptor associated with the sub-graph interface.

7. The method of claim 6 , wherein the second dataflow graph specification includes at least one indicates that a descriptor associated with a fourth component is identical to the descriptor associated with the third component.

8. The method of claim 7 , wherein the determined direction corresponds to the outward transfer of the value of the descriptor from the flow junction on the sub-graph interface to the third component.

9. The method of claim 8 , wherein the descriptor is provided to the flow junction from the first component.

10. The method of claim 9 , wherein the descriptor is provided to the fourth component from the third component.

11. The method of claim 1 , wherein the descriptor describes a data characteristic of data transferred between a port of the first component and a port of the second component.

12. The method of claim 11 , wherein the data characteristic includes a format of fields of records within the transferred data.

13. The method of claim 1 , wherein the descriptor describes a computational characteristic of a component of at least one of the first dataflow graph specification or the second dataflow graph specification.

14. The method of claim 13 , wherein the computational characteristic includes a degree of parallelism of execution of a computation represented by the component.

15. The method of claim 1 , wherein a first dataflow graph, based on the first dataflow graph specification, is encrypted.

16. The method of claim 1 , wherein a second dataflow graph, based on the second dataflow graph specification, is encrypted.

17. The method of claim 1 , wherein a first dataflow graph, based on the first dataflow graph specification has the two or more components of the first dataflow graph specification distributed among a plurality of sequentially executing phases in the first dataflow graph.

18. The method of claim 17 , wherein the plurality of sequentially executing phases is a first plurality of sequentially executing phases, and a second dataflow graph, based on the second dataflow graph specification has a plurality of components including the third component, with the plurality of components and the sub-graph interface distributed among a second plurality of sequentially executing phases in the second dataflow graph.

19. The method of claim 18 , further includes
preparing the first dataflow graph and the second dataflow graph for execution by determining a number of phases for inclusion in the dataflow graph based on the first plurality of sequentially executing phases and the second plurality of sequentially executing phases.

20. The method of claim 1 , wherein the descriptor is in the form of metadata, the at least one sub-graph interface includes a plurality of flow junctions, which includes the flow junction, and with processing further including
performing edit-time metadata propagation for each of the first dataflow graph specification and the second dataflow graph specification, with the edit-time metadata propagation including
associating values of descriptors including the descriptor with ports of the first dataflow graph specification and the second dataflow graph specification and the plurality of flow junctions.

21. The method of claim 20 , wherein the processing further includes linking the first dataflow graph specification to the second dataflow graph specification in place of the sub-graph interface after identifying the association between the sub-graph interface and the first dataflow graph specification.

22. The method of claim 21 , wherein the processing further includes
performing link-time metadata propagation, after the edit-time metadata propagation is complete for each of a first dataflow graph and a second dataflow graph, the link-time metadata propagation including transferring the value of the descriptor from the first flow junction.

23. The method of claim 22 , wherein the flow junction is a first flow junction, and the edit-time metadata propagation for the first dataflow graph specification includes propagating a placeholder value associated with a second flow junction on the sub-graph interface within the first dataflow graph specification.

24. The method of claim 23 , wherein the link-time metadata propagation includes resolving the placeholder value by replacing the placeholder value with a value of a descriptor of a data or computational characteristic associated with the second flow junction.

25. The method of claim 1 , wherein determining the direction associated with a transfer of the value of the descriptor includes de

Assignees

• Ab Initio Technology Llc

Inventors

• Larson Brond
• Bay Paul
• Bromley H Mark

Classifications

• G06F8/10

  Requirements analysis; Specification techniques · CPC title

• G06F8/34 Primary

  Graphical or visual programming · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title

• G06F9/4494

  data driven · CPC title

• G06F8/20

  Software design · CPC title