Evaluating dataflow graph characteristics

Patent metadata

Field	Value
Publication number	US-9727438-B2
Application number	US-201113217778-A
Country	US
Kind code	B2
Filing date	Aug 25, 2011
Priority date	Aug 25, 2010
Publication date	Aug 8, 2017
Grant date	Aug 8, 2017

Abstract

One or more expressions are evaluated that represent one or more characteristics of a dataflow graph that includes vertices representing data processing components connected by links representing flows of work elements between the components. A request is received by a computing system to evaluate the one or more expressions that include one or more operations on one or more variables; and the one or more expressions are evaluated by the computing system. The evaluating includes: defining a data structure that includes one or more fields, collecting, during execution of the dataflow graph, tracking information associated with one or more components of the dataflow graph, storing values associated with the tracking information in the one or more fields, and replacing one or more variables of the one or more expressions with the values stored in the one or more fields to compute a result of evaluating the one or more expressions.

First claim

What is claimed is:

1. A method performed by a computing system having a processor and a memory, the computing system for determining a performance characteristic of a dataflow graph that includes vertices representing data processing components connected by links representing flows of work elements between the data processing components, the method including:
receiving, by the processor, data representing one or more expressions that represent a characteristic of the dataflow graph,
defining, in the memory, fields having field types for an in-memory data structure, the defining being based on analysis of the one or more expressions,
accessing, by the processor, tracking information produced during execution of the dataflow graph, the tracking information associated with one or more of the data processing components of the dataflow graph,
storing, in the in-memory data structure, one or more values determined based on a portion of the tracking information in a corresponding field of the in-memory data structure, wherein the portion of the tracking information is extracted according to the field types for the defined fields for the in-memory data structure, and
determining, by the processor, one or more of an amount of data being processed by the dataflow graph and an amount of resources being consumed during the processing by the dataflow graph by:
selecting one or more values stored in the fields of the in-memory data structure, and
evaluating the one or more expressions by replacing each of one or more variables of the one or more expressions with a corresponding one of the values stored in the fields of the in-memory data structure.

2. The method of claim 1 , wherein the evaluating is performed while the dataflow graph is being executed.

3. The method of claim 1 , wherein the tracking information is further associated with one or more components of one or more subgraphs of the dataflow graph.

4. The method of claim 3 , wherein at least a first variable of the one or more variables is expressed in the one or more expressions as a term that includes a reference to a first subgraph one of the one or more subgraphs.

5. The method of claim 4 , wherein the first variable includes a reference to a first component within the first subgraph.

6. The method of claim 3 , wherein at least one of the one or more variables corresponds to a characteristic of one or more of (i) the dataflow graph, (ii) the one or more subgraphs, and (iii) the one or more components.

7. The method of claim 1 , wherein evaluating the one or more expressions includes evaluating the one or more expressions according to a predefined condition.

8. The method of claim 7 , wherein the predefined condition includes a detection of a trigger event.

9. The method of claim 8 , wherein the trigger event includes a determination that one or more processing resources are scarce relative to demand for the one or more processing resources.

10. The method of claim 1 , wherein defining the in-memory data structure includes traversing the dataflow graph to identify one or more of the field type and a number of the fields.

11. The method of claim 10 , wherein traversing the dataflow graph includes traversing the portions of the dataflow graph associated with the one or more expressions.

12. The method of claim 1 , including filtering the tracking information based at least in part on the one or more expressions.

13. The method of claim 1 , further including comparing a result of the evaluating against one or more threshold values.

14. The method of claim 13 , further including generating a notification based on the comparing.

15. The method of claim 14 , including generating the notification when a result of evaluating the one or more expressions exceeds a threshold value indicative of the allowable performance characteristic.

16. The method of claim 14 , further including executing one or more automated processes based at least in part on the notification.

17. The method of claim 1 wherein receiving the one or more expressions includes receiving the one or more expressions from a user interface.

18. The method of claim 1 , further including providing a set of one or more predefined expressions that includes the one or more expressions, and in which receiving the one or more expressions includes receiving a selection of one or more of the predefined expressions.

19. The method of claim 1 , wherein the evaluating further includes periodically evaluating the one or more expressions.

20. The method of claim 19 , further including generating a histogram associated with one or more characteristics of the dataflow graph using two or more results of evaluating the one or more expressions.

21. The method of claim 19 , further including updating stored historical data based at least in part on results of periodically evaluating the one or more expressions.

22. The method of claim 19 , wherein at least one of the one or more variables depends on a value that was stored in the historical data before a request to evaluate the one or more expressions was received by the computing system.

23. The method of claim 1 , wherein storing the one or more values includes storing the one or more values in a vector within the corresponding field.

24. The method of claim 23 , wherein each of the one or more values is associated with a different respective value of a particular parameter, and each of the one or more values is a value of a characteristic that is aggregated over tracking information associated with the respective value of the particular parameter.

25. The method of claim 1 , in which defining the fields for the in-memory data structure includes defining the fields for the in-memory data structure based on a lexical analysis of the one or more expressions.

26. The method of claim 1 , including receiving a request to evaluate the one or more expressions.

27. The method of claim 1 , including generating the notification when the result of evaluating the one or more expressions falls outside of a range of values indicative of an allowable performance characteristic.

28. The method of claim 1 , wherein the in-memory data structure is stored in volatile memory.

29. The method of claim 1 , wherein the in-memory data structure is stored in non-volatile memory.

30. The method of claim 1 , wherein the amount of data being processed includes one or more of a number of records and a number of bytes.

31. The method of claim 1 , wherein the amount of resources being consumed includes CPU time being used by a data processing component of the dataflow graph.

32. The method of claim 1 , wherein the amount of resources being consumed includes a latency associated with a data processing component of the dataflow graph.

33. The method of claim 1 , including extracting the portion of the tracking information according to the field types for the defined fields for the in-memory data structure.

34. A non-transitory computer-readable storage medium storing a computer program for determining a performance characteristic of a dataflow graph that includes vertices representing data processing components connected by links representing flows of work elements between the data processing components, the computer program including i

Assignees

• Buxbaum Mark
• Feinhaus Dima V
• Wakeling Tim
• Ab Initio Technology Llc

Inventors

• Buxbaum Mark
• Feinhaus Dima V
• Wakeling Tim

Classifications

• G06F11/3476 Primary

  Data logging (G06F11/14, G06F11/2205 take precedence) · CPC title

• G06F9/50

  Allocation of resources, e.g. of the central processing unit [CPU] · CPC title

• G06F11/34 Primary

  Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation {; Recording or statistical evaluation of user activity, e.g. usability assessment} · CPC title