Managing interfaces for sub-graphs

Patent metadata

Field	Value
Publication number	US-10318252-B2
Application number	US-201715852452-A
Country	US
Kind code	B2
Filing date	Dec 22, 2017
Priority date	Dec 5, 2013
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

Specifying a dataflow graph includes: rendering, in a first user interface, a representation of a first dataflow graph. At least one component represents a computation associated with at least one of data flowing into an input port or data flowing out of an output port. A link between components is rendered based on user input indicating a flow of data. In a second user interface, a representation of a sub-graph of the first dataflow graph is rendered. A first flow junction represents a connection between: (1) a flow of data outside the sub-graph interface, and (2) a flow of data inside the sub-graph interface. A link is rendered between the first flow junction and a second flow junction based on user input indicating a relationship between a first descriptor and a second descriptor.

First claim

What is claimed is:

1. A method for managing different dataflow graph specifications, the method including:
storing in a storage system a plurality of dataflow graph specifications, including:
a first dataflow graph specification that specifies two or more components connected by links representing flows of data between components, including a first component and a second component, with at least one component representing a computation applied to data flowing into an input port of the component, and at least one component representing a computation providing data flowing out of an output port of the component, and
a second dataflow graph specification that specifies at least a third component, and at least one sub-graph interface, where the sub-graph interface includes at least one flow junction representing a connection between: (1) a flow of data outside the sub-graph interface to or from a port of the third component, and (2) a flow of data inside the sub-graph interface to or from a port of a component of the first dataflow graph specification; and

processing, using at least one processor, information including the first dataflow graph specification and the second dataflow graph specification, to generate a combined dataflow graph for execution, the processing including:
identifying an association between the sub-graph interface and the first dataflow graph specification,
for at least a first flow junction on the sub-graph interface, determining a direction associated with transferring a value of a descriptor of a data or computational characteristic, and
transferring a value of a descriptor of a data or computational characteristic in an inward direction from the first flow junction to a component specified by the first dataflow graph specification if the determined direction indicates an inward direction, or in an outward direction from the first flow junction to a component specified by the second dataflow graph specification if the determined direction indicates an outward direction.

2. The method of claim 1 , wherein the first dataflow graph specification is dynamically linked into the second dataflow graph specification during dynamic linking that before execution of the combined dataflow graph.

3. The method of claim 2 , wherein descriptors of data or computational characteristics associated with flow junctions are propagated during metadata propagation for the second dataflow graph specification that occurs after the dynamic linking and before the execution of the combined dataflow graph.

4. The method of claim 3 , wherein the flow of data outside the sub-graph interface and the flow of data inside the sub-graph interface are generated as a single flow during execution of the combined dataflow graph.

5. The method of claim 1 , wherein the first dataflow graph specification includes at least one indicator that indicates that a descriptor associated with the first component is identical to a descriptor associated with the second component.

6. The method of claim 5 , wherein the determined direction corresponds to an inward direction from the first flow junction to the second component.

7. The method of claim 6 , wherein the value of the descriptor is provided to the first flow junction from the third component, before being transferred in an inward direction.

8. The method of claim 7 , wherein the value of the descriptor is provided to the first component from the second component, after being transferred in an inward direction.

9. The method of claim 1 , wherein the second dataflow graph specification includes at least one indicator that indicates that a descriptor associated with the third component is identical to a descriptor associated with the sub-graph interface.

10. The method of claim 9 , wherein the second dataflow graph specification includes at least one indicator that indicates that a descriptor associated with a fourth component is identical to a descriptor associated with the third component.

11. The method of claim 10 , wherein the determined direction corresponds to an outward direction from the first flow junction to the third component.

12. The method of claim 11 , wherein the value of the descriptor is provided to the first flow junction from the first component, before being transferred in an outward direction.

13. The method of claim 12 , wherein the value of the descriptor is provided to the fourth component from the third component, after being transferred in an outward direction.

14. The method of claim 1 , wherein the descriptor describes a data characteristic of data transferred between an output port of the first component and an input port of the second component.

15. The method of claim 14 , wherein the data characteristic includes a format of fields of records within the transferred data.

16. The method of claim 1 , wherein the descriptor describes a computational characteristic of a component of at least one of the first dataflow graph specification or the second dataflow graph specification.

17. The method of claim 16 , wherein the computational characteristic includes a degree of parallelism of execution of a computation represented by the component.

18. The method of claim 1 wherein the first dataflow graph is encrypted.

19. The method of claim 17 wherein the two or more components of the first dataflow graph are distributed among a first plurality of sequentially executing phases in the first dataflow graph.

20. The method of claim 19 wherein the second dataflow graph includes a plurality of components, the plurality of components and the sub-graph interface distributed among a second plurality of sequentially executing phases in the second dataflow graph.

21. The method of claim 19 wherein preparing the dataflow graph for execution includes determining a number of phases for inclusion in the dataflow graph based on the first plurality of sequentially executing phases and the second plurality of sequentially executing phases.

22. The method of claim 1 wherein the sub-graph interface is specified based on a rendered representation of the sub-graph interface that includes a link between the first flow junction of the sub-graph interface and a second flow junction of the sub-graph interface, where the link is rendered based on user input indicating a relationship between: (1) a first descriptor of a data or computational characteristic associated with the first flow junction of the sub-graph interface, and (2) a second descriptor of a data or computational characteristic associated with the second flow junction of the sub-graph interface.

23. The method of claim 22 , wherein the relationship corresponds to the first descriptor being identical to the second descriptor.

24. Software stored in a non-transitory form on a computer-readable medium, for specifying a dataflow graph, the software including instructions for causing a computing system to:
store in a storage system a plurality of dataflow graph specifications, including:
a first dataflow graph specification that specifies two or more components connected by links representing flows of data between components, including a first component and a second component, with at least one component representing a computation applied to data flowing into an input port of the component, and at least one component representing a computation providing data flowing out of an output port of the component, and
a second dataflow graph specification that spe

Assignees

• Ab Initio Technology Llc

Inventors

• Bay Paul
• Bromley H Mark
• Larson Brond
• Stevens Daniell Garrick
• Tsvetanov Tsvetan

Classifications

• G06F8/10

  Requirements analysis; Specification techniques · CPC title

• G06F8/34 Primary

  Graphical or visual programming · CPC title

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title

• G06F9/4494

  data driven · CPC title

• G06F8/20

  Software design · CPC title