Dynamic graph performance monitoring

Patent metadata

Field	Value
Publication number	US-9507682-B2
Application number	US-201213678921-A
Country	US
Kind code	B2
Filing date	Nov 16, 2012
Priority date	Nov 16, 2012
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for dynamic graph performance monitoring. One of the methods includes receiving multiple units of work that each include one or more work elements. The method includes determining a characteristic of the first unit of work. The method includes identifying, by a component of the first dataflow graph, a second dataflow graph from multiple available dataflow graphs based on the determined characteristic, the multiple available dataflow graphs being stored in a data storage system. The method includes processing the first unit of work using the second dataflow graph. The method includes determining one or more performance metrics associated with the processing.

First claim

What is claimed is:

1. A method for processing data in a data processing system, wherein a first dataflow graph is executed, the method including:
receiving multiple units of work that each include one or more records;
determining a characteristic of a first unit of work based on data included in the one or more records of the first unit of work;
identifying, by a component of the first dataflow graph, a second dataflow graph from multiple available dataflow graphs based on the determined characteristic, the multiple available dataflow graphs being stored in a data storage system;
processing the first unit of work using the second dataflow graph based on loading the second dataflow graph into the first dataflow graph;
receiving, by the first dataflow graph, one or more values of one or more performance metrics associated with the processing of the first unit of work using the loaded second dataflow graph;
outputting, by the first dataflow graph to a performance monitoring module, the one or more received values of the one or more performance metrics associated with the processing of the first unit of work using the loaded second dataflow graph, wherein the performance monitoring module receives a plurality of values of one or more performance metrics output by a plurality of dataflow graphs;
selecting, from among the plurality of values of the one or more performance metrics output by the plurality of dataflow graphs, values of one or more performance metrics associated with an identifier for the second dataflow graph; and
aggregating the values selected for the one or more performance metrics associated with the identifier for the second dataflow graph.

2. The method of claim 1 , wherein the second dataflow graph is compiled independent of the first dataflow graph.

3. The method of claim 1 , wherein the processing further includes storing the one or more values of the one or more performance metrics associated with the processing of the first unit of work using the loaded second dataflow graph in an in-memory data store.

4. The method of claim 3 , wherein the processing further includes transferring the stored one or more values of the one or more performance metrics to a persistent data store.

5. The method of claim 1 , wherein the values selected include first values and wherein the method further includes aggregating second values of the one or more performance metrics output by the plurality of dataflow graphs based on an identifier associated with the first dataflow graph.

6. The method of claim 1 , further including displaying the one or more performance metrics to a user or displaying the aggregated values of the one or more performance metrics to the user.

7. A non-transitory computer-readable storage medium storing a computer program for processing data in a data processing system, wherein a first dataflow graph is executed, the computer program including instructions for causing a computing system to:
receive multiple units of work that each include one or more records;
determine a characteristic of a first unit of work based on data included in the one or more records of the first unit of work;
identify, by a component of the first dataflow graph, a second dataflow graph from multiple available dataflow graphs based on the determined characteristic, the multiple available dataflow graphs being stored in a data storage system;
process the first unit of work using the second dataflow graph based on loading the second dataflow graph into the first dataflow graph;
receive, by the first dataflow graph, one or more values of one or more performance metrics associated with the processing of the first unit of work using the loaded second dataflow graph;
output, by the first dataflow graph to a performance monitoring module, the one or more received values of the one or more performance metrics associated with the processing of the first unit of work using the loaded second dataflow graph, wherein the performance monitoring module receives a plurality of values of one or more performance metrics output by a plurality of dataflow graphs;
select, from among the plurality of values of the one or more performance metrics output by the plurality of dataflow graphs, values of one or more performance metrics associated with an identifier for the second dataflow graph; and
aggregate the values selected for the one or more performance metrics associated with the identifier for the second dataflow graph.

8. The medium of claim 7 , wherein the second dataflow graph is compiled independent of the first dataflow graph.

9. The medium of claim 7 , wherein the computer program further includes instructions for causing a computing system to store the one or more values of the one or more performance metrics associated with the processing of the first unit of work using the loaded second dataflow graph in an in-memory data store.

10. The medium of claim 9 , wherein the computer program further includes instructions for causing a computing system to transfer the stored one or more values of the one or more performance metrics to a persistent data store.

11. The medium of claim 7 , wherein the values selected include first values and wherein the instructions to aggregate include instructions to aggregate second values of the one or more performance metrics output by the plurality of dataflow graphs based on an identifier associated with the first dataflow graph.

12. The medium of claim 7 , wherein the computer program further includes instructions for causing a computing system to display the one or more performance metrics to a user or to display the aggregated values of the one or more performance metrics to the user.

13. A computing system for processing data, the computing system including:
an input device or port configured to receive multiple units of work that each include one or more records; and
at least one processor configured to process data, the processing including:
receiving multiple units of work that each include one or more records;
determining a characteristic of a first unit of work based on data included in the one or more records of the first unit of work;
identifying, by a component of a first dataflow graph, a second dataflow graph from multiple available dataflow graphs based on the determined characteristic, the multiple available dataflow graphs being stored in a data storage system;
processing the first unit of work using the second dataflow graph based on loading the second dataflow graph into the first dataflow graph;
receiving, by the first dataflow graph, one or more values of one or more performance metrics associated with the processing of the first unit of work using the loaded second dataflow graph;
outputting, by the first dataflow graph to a performance monitoring module, the one or more received values of the one or more performance metrics associated with the processing of the first unit of work using the loaded second dataflow graph, wherein the performance monitoring module receives a plurality of values of one or more performance metrics output by a plurality of dataflow graphs;
selecting, from among the plurality of values of the one or more performance metrics output by the plurality of dataflow graphs, values of one or more performance metrics associated with an identifier for the second dataflow graph; and
aggregating the values selected for the one or more performance metrics associated with the identifier for the second dataflow graph.

14. The system of claim 13 , wherein the second dataflow graph is compiled independent of the first dataflow graph.

15. The syste

Assignees

• Ab Initio Technology Llc

Inventors

• Buxbaum Mark
• Mulligan Michael G
• Wakeling Tim
• Atterbury Matthew Darcy

Classifications

• G06F11/3082

  the data filtering being achieved by aggregating or compressing the monitored data · CPC title

• G06F11/3003 Primary

  Monitoring arrangements specially adapted to the computing system or computing system component being monitored · CPC title

• G06F11/3419

  by assessing time · CPC title

• G06F11/3404

  for parallel or distributed programming · CPC title

• G06Q30/0201

  Market modelling; Market analysis; Collecting market data · CPC title