Managing parameter sets

Patent metadata

Field	Value
Publication number	US-11210086-B2
Application number	US-201916389148-A
Country	US
Kind code	B2
Filing date	Apr 19, 2019
Priority date	Jul 18, 2014
Publication date	Dec 28, 2021
Grant date	Dec 28, 2021

Abstract

Managing sets of parameter values includes: receiving a plurality of sets of parameter values for a generic computer program, and processing log entries associated with executions of instances of the generic computer program, each instance associated with one or more parameter values. The processing includes: analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class or a second class; processing a log entry associated with an execution of a first instance of the generic computer program to form a particular set of parameter values; and determining whether to add the particular set of parameter values to the plurality of sets of parameter values based on a comparison of a first identifier for the particular set of parameter values to identifiers for at least some of the sets of parameter values.

First claim

What is claimed is:

1. A method for managing sets of parameter values, the method including:
receiving a generic computer program;
receiving a first set of parameter values;
generating an executable instance of the generic computer program by instantiating the generic computer program according to the first set of parameter values;
receiving data from one or more data sets;
executing the executable instance of the generic computer program to process at least some of the received data;
generating a log entry for the executable instance of the generic computer program, the log entry including at least some of the parameter values of the first set of parameter values;
storing the log entry in a log data structure;
receiving the log entry, including retrieving the stored log entry from the log data structure;
processing the received log entry to form a particular set of parameter values, wherein the processing includes extracting the at least some of the parameter values of the first set of parameter values from the log entry and forming the particular set of parameter values from the extracted at least some of the parameter values; and
determining whether to add the particular set of parameter values to a plurality of preexisting sets of parameter values based on a comparison of the particular set of parameter values to at least some sets of parameter values of the plurality of preexisting sets of parameter values, wherein the preexisting sets of parameter values represent data lineage relationships between inputs and outputs of a plurality of computer programs.

2. The method of claim 1 wherein the comparison includes comparing a first identifier for the particular set of parameter values to identifiers for the at least some of the preexisting sets of parameter values of the plurality of preexisting sets of parameter values including:
determining the first identifier based on the particular set of parameter values and an identifier of the generic computer program;
determining a plurality of second identifiers, one for each preexisting set of parameter values of the at least some of the preexisting sets of parameter values; and
comparing the first identifier to each second identifier of the plurality of second identifiers to determine whether the first identifier and any of the second identifiers match.

3. The method of claim 2 wherein determining whether to add the particular set of parameter values to the plurality of preexisting sets of parameter values includes determining to add the particular set of parameter values to the plurality of preexisting sets of parameter values if none of the second identifiers match the first identifier.

4. The method of claim 2 wherein determining the first identifier includes computing an identification string from contents of the particular set of parameter values, and determining the second identifiers includes computing identification strings from the contents of the at least some of the preexisting sets of parameter values of the plurality of preexisting sets of parameter values.

5. The method of claim 2 wherein determining the first identifier includes forming a concatenation of one or more of the identifier of the generic computer program, name-value pairs of the particular set of parameter values, a function prototype of the generic computer program, and a project scope for the executable instance of the generic computer program.

6. The method of claim 2 wherein determining the first identifier includes applying a data mapping function to one or more of the identifier of the generic computer program, name-value pairs of the particular set of parameter values, a function prototype of the generic computer program, and a project scope for the executable instance of the generic computer program.

7. The method of claim 6 wherein the data mapping function includes a hash function.

8. The method of claim 1 further including analyzing the generic computer program to classify each of one or more parameters associated with the generic computer program as a member of either a first class of parameters or a second class of parameters.

9. The method of claim 8 wherein processing the log entry to form the particular set of parameter values further includes:
including, in the particular set, any extracted parameter values occurring in the log entry and classified as members of the first class, and
excluding, from the particular set, any extracted parameter values occurring in the log entry and classified as members of the second class.

10. The method of claim 8 wherein, for each of the parameter one or more parameters, the analyzing includes either automatically classifying the parameter or accepting a user-defined classification for the parameter.

11. The method of claim 10 wherein automatically classifying the parameter includes initially classifying the parameter as belonging to the first class of parameters, determining a number of unique values of the parameter in a plurality of executions of instances of the generic computer program, and re-classifying the parameter as belonging to the second class of parameters if the number of unique values of the parameter exceeds a predetermined threshold.

12. The method of claim 10 wherein automatically classifying the parameter includes initially classifying the parameter as belonging to the first class of parameters, determining whether or not changes in values of the parameter in a plurality of executions of instances of the generic computer program affect data lineage associated with the generic computer program, and re-classifying the parameter as belonging to the second class of parameters if changes in values of the parameter do not affect the data lineage.

13. The method of claim 8 wherein the first class of parameters includes parameters that affect a logical operation of the generic computer program and the second class of parameters includes parameters that do not affect the logical operation of the generic computer program.

14. The method of claim 1 wherein the generic computer program is specified as a dataflow graph that includes nodes representing data processing operations and links between the nodes representing flows of data elements between the data processing operations.

15. The method of claim 1 further including forming an association between the log entry and the particular set of parameter values.

16. The method of claim 1 wherein the log entry includes a log entry of an execution command used to instantiate the executable instance of the generic computer program including one or more parameter values supplied as arguments to the execution command.

17. The method of claim 16 wherein the log entry further includes one or more of an indication of a project that the executable instance executed in, an indication of internal parameters for the executable instance, and an indication of environmental settings, global variables, and configuration variables used by the executable instance.

18. The method of claim 1 further including processing an overall plurality of sets of parameter values for a plurality of generic computer programs and an overall plurality of log entries associated with executions of instances of at least some of the plurality of generic computer programs to form a data lineage report, wherein the overall plurality of sets of parameter values includes the particular set of parameter values, and the overall plurality of log entries associated with executions of instances of the at least some of the plurality of generic computer pro

Assignees

• Ab Initio Technology Llc

Inventors

• Bach Edward
• Oberdorf Richard
• Larson Brond

Classifications

• G06F16/285

  Clustering or classification · CPC title

• G06F8/34

  Graphical or visual programming · CPC title

• G06F8/10

  Requirements analysis; Specification techniques · CPC title

• G06F9/4494

  data driven · CPC title

• G06F16/2282

  Tablespace storage structures; Management thereof · CPC title