Managing data set objects

Patent metadata

Field	Value
Publication number	US-9977659-B2
Application number	US-201113281039-A
Country	US
Kind code	B2
Filing date	Oct 25, 2011
Priority date	Oct 25, 2010
Publication date	May 22, 2018
Grant date	May 22, 2018

Abstract

Managing data set objects for graph-based data processing includes: storing a group of one or more data set objects in a data storage system, the data set objects each representing a respective data set; and generating an association between at least a first data set object in the group and at least a first node of a dataflow graph for processing data in a data processing system, the first node representing a source or sink of data in a flow of data represented by a link in the dataflow graph, and the first data set object including a plurality of modes in which different transformational logic is applied to data processed by the first node.

First claim

What is claimed is:

1. A method for managing data set objects for use in connection with a dataflow graph processing data in a data processing system, the method including:
storing a group of one or more data set objects in a data storage system, each data set object in the group representing a respective data set, the group including a first data set object representing a first data set, the first data set object having a plurality of modes, each of the plurality of modes corresponding to respective transformational logic, the plurality of modes including at least:
a first mode for reading data records from the first data set, the first mode being associated with first transformational logic to be applied to data records read from the first data set, and
a second mode for writing data records to the first data set, the second mode being associated with second transformational logic to be applied to data records before they are written to the first data set, wherein the first transformational logic is different from the second transformational logic;

accessing information specifying the dataflow graph, the dataflow graph comprising a plurality of nodes and a plurality of links connecting the plurality of nodes, the plurality of links representing flows of data among nodes in the plurality of nodes;
identifying the first data set object from the group of one or more data set objects;
identifying a mode from the plurality of modes for the first data set object;
adding, based on the accessed information specifying the dataflow graph, a first node to the plurality of nodes of the dataflow graph; and
generating an association between the first data set object and the first node of the dataflow graph such that data processed by the first node is processed in accordance with transformational logic associated with the identified mode.

2. The method of claim 1 , wherein generating the association between the first data set object and the first node includes:
connecting the first node to at least one other node in the dataflow graph using a link.

3. The method of claim 1 , wherein one of the plurality of modes of the first data set object is represented by a second dataflow graph different from the dataflow graph.

4. The method of claim 1 , wherein the plurality of modes for the first data set object includes multiple modes for reading data records to the dataset, the multiple modes including the first mode, wherein each of the multiple modes is associated with respective transformational logic to be applied to data records read from the first data set.

5. The method of claim 1 , wherein the plurality of modes for the first data set object includes multiple modes for writing data records to the dataset, the multiple modes including the second mode, wherein each of the multiple modes is associated with respective transformational logic to be applied to data records before they are written to the first data set.

6. The method of claim 1 , wherein data records in the first data set are stored in an XML format and the first transformational logic, when executed, transforms the data records from the XML format into a series of sequential data records.

7. The method of claim 1 , wherein the second transformational logic, when executed, transforms a series of sequential data records into an XML format.

8. The method of claim 1 , wherein identifying the mode from the plurality of modes is performed based on user input provided via a graphical user interface, the user input indicating a selection of the mode.

9. The method of claim 1 , wherein identifying the first data set object comprises:
receiving, in a graphical user interface displaying a catalog of data set objects in the group and a canvas, first user input indicating a selection of the first data set object and its placement at a location on the canvas.

10. A non-transitory computer-readable storage medium storing a computer program, that when executed by at least one processor, causes the at least one processor to perform a method for managing data set objects for use in connection with a dataflow graph processing data in a data processing system, the method comprising:
storing a group of one or more data set objects in a data storage system, each data set object in the group representing a respective data set, the group including a first data set object representing a first data set, the first data set object having a plurality of modes, each of the plurality of modes corresponding to respective transformational logic, the plurality of modes including at least:
a first mode for reading data records from the first data set, the first mode being associated with first transformational logic to be applied to data records read from the first data set, and
a second mode for writing data records to the first data set, the second mode being associated with second transformational logic to be applied to data records before they are written to the first data set, wherein the first transformational logic is different from the second transformational logic;

accessing information specifying the dataflow graph, the dataflow graph comprising a plurality of nodes and a plurality of links connecting the plurality of nodes, the plurality of links representing flows of data among nodes in the plurality of nodes;
identifying the first data set object from the group of one or more data set objects;
identifying a mode from the plurality of modes for the first data set object;
adding, based on the accessed information specifying the dataflow graph, a first node to the plurality of nodes of the dataflow graph; and
generating an association between the first data set object and the first node of the dataflow graph such that data processed by the first node is processed in accordance with transformational logic associated with the identified mode.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the plurality of modes for the first data set object includes multiple modes for reading data records to the dataset, the multiple modes including the first mode, wherein each of the multiple modes is associated with respective transformational logic to be applied to data records read from the first data set.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the plurality of modes for the first data set object includes multiple modes for writing data records to the dataset, the multiple modes including the second mode, wherein each of the multiple modes is associated with respective transformational logic to be applied to data records before they are written to the first data set.

13. The non-transitory computer-readable storage medium of claim 10 , wherein data records in the first data set are stored in an XML format and the first transformational logic, when executed, transforms the data records from the XML format into a series of sequential data records.

14. The non-transitory computer-readable storage medium of claim 10 , wherein identifying the mode from the plurality of modes is performed based on user input provided via a graphical user interface, the user input indicating a selection of the mode.

15. The non-transitory computer-readable storage medium of claim 10 , wherein identifying the first data set object comprises:
receiving, in a graphical user interface displaying a catalog of data set objects in the group and a canvas, first user input indicating a selection of the first data set object and its placement at a location on the canvas.

16. A computing system for managing data set objects for use in conne

Assignees

• Larson Brond
• Shapiro Richard A
• Stanfill Craig W
• Weiss Adam Harris
• Ab Initio Technology Llc

Inventors

• Larson Brond
• Shapiro Richard A
• Stanfill Craig W
• Weiss Adam Harris

Classifications

• G06F8/34 Primary

  Graphical or visual programming · CPC title

• G06F16/23

  Updating · CPC title

• G06F17/00 Primary

  Digital computing or data processing equipment or methods, specially adapted for specific functions (information retrieval, database structures or file system structures therefor G06F16/00) · CPC title