Unit of work based incremental data processing

Patent metadata

Field	Value
Publication number	US-9454557-B2
Application number	US-201414158964-A
Country	US
Kind code	B2
Filing date	Jan 20, 2014
Priority date	Aug 29, 2013
Publication date	Sep 27, 2016
Grant date	Sep 27, 2016

Abstract

Systems, methods, and other embodiments associated with a unit of work for incremental data processing are described. In one embodiment, a method includes selecting a unit of work key for a program that processes records in a source table. The unit of work key is selected such that modifications to a record having a certain unit of work key value will not affect the program's processing of records having a different unit of work key value. The selected unit of work key is associated with the program. When a record in the source table is created, modified, or deleted, a unit of work key value is identified for the record and a selected set of records having the identified unit of work key value is provided to the program. Thus, records in the data source that do not have the identified unit of work key values are not provided to the program.

First claim

What is claimed is:

1. A non-transitory computer storage medium storing computer-executable instructions that when executed by a processor of a computing device cause the processor to perform:
identifying a unit of work key that is predefined for a program, where the program processes data from a data source, and further where the data source comprises one or more source tables and the unit of work key comprises one or more columns in the source tables;
accessing a unit of work driver table that records unit of work key values for records in the data source that have been created, modified, or deleted in any source table since a last time the program was executed to determine a unique set of unit of work key values;
selecting a set of records by creating a view query that selects, for a view, records from each source table that have the unit of work key values in the unit of work driver table; and
providing the view of the selected records to the program.

2. The non-transitory computer storage medium of claim 1 , where the program accesses the view as source data.

3. The non-transitory computer storage medium of claim 1 , further comprising instructions that when executed cause the processor to populate the unit of work driver table by:
accessing a program tracking table that records a last source modification time corresponding to a timestamp of the data source when the program was last executed; and
identifying the unique set of respective unit of work key values for records in the data source having a last modification time that is later than the recorded last source modification time.

4. The non-transitory computer storage medium of claim 1 , where the unit of work driver table comprises a source tracking table populated by a data management program that manages the data source, and where the source tracking table records unit of work key values for records that have been created, modified, or deleted since a last time the program was executed.

5. The non-transitory computer storage medium of claim 1 , where the view query selects all records having a unit of work key in common with any record that has been created, modified, or deleted since a last time the program was executed.

6. The non-transitory computer storage medium of claim 1 , where the program processes the selected records and populates a target table, where the instructions further comprise instructions for causing the processor to delete a record from the target table when:
the record has a unit of work key value in the set of unique work key values; and
the record has not been created, modified, or refreshed by execution of the program in the target table.

7. The non-transitory computer storage medium of claim 1 where the instructions for identifying comprise instructions for accepting an input of the unit of work key for the program from a user or an external function that selects the unit of work key for the program.

8. The non-transitory computer storage medium of claim 1 , where the unit of work key comprises a first key and a second key, where the program inputs records from at least a first table comprising the first key and the second key and a second table comprising the first key and not the second key, and further comprising instructions for:
accessing the unit of work driver table to determine:
a unique set of unit of work first key value and second key value pairs for records in the first table that have been created, modified, or deleted since a last time the program was executed; and
a unique set of unit of work first key values for records in the second table that have been created, modified, or deleted since the last time the program was executed; and

creating a view query that:
selects records in the first table that have the first key value and second key value pairs determined in the first table or that have the first key values determined in the second table;
selects records in the second table that have the first key values determined in the first table or that have the first key values determined in the second table.

9. A computing system, comprising:
a processor connected to a memory;
a non-transitory computer storage medium connected to a data communication path with at least the processor and the memory;
an incremental processing logic stored in the non-transitory computer storage medium and including instructions executable by the processor to cause the processor to identify a unit of work key that is predefined for a program, where the program is executable by the computing system to process data from a data source, and further where the data source comprises one or more source tables and the unit of work key comprises one or more columns in the source tables;
a unit of work tracking logic stored in the non-transitory computer storage medium and including instructions executable by the processor to cause the processor to:
determine a unique set of unit of work key values for records in the data source that have been created, modified, or deleted since a last time the program was executed; and
populate a unit of work driver table with the unique set of unit of work key values;

a view logic stored in the non-transitory computer storage medium and including instructions executable by the processor to cause the processor to:
create a view that accesses the unit of work driver table to select a set of records from the data source that have the determined unit of work key values; and
where the program, during execution of the program, reads input from the view to perform processing of data from the data source.

10. The computing system of claim 9 , where the unit of work driver table comprises
one or more source tracking tables populated by a data management program that manages the data source, where the source tracking tables record respective unique unit of work key values for records in the data source
that have been created, modified, or deleted since the last time the program was executed.

11. The computing system of claim 9 , where the incremental processing logic comprises:
a program tracking logic configured to access a program tracking table that records a last execution time of the program; and
and where the unit of work tracking logic is configured to identify respective unit of work key values for respective records in the data source having a last modification time that is later than the recorded last execution time and populate the unit of work driver table with the identified unit of work key values.

12. The computing system of claim 9 , where the program processes the selected set of records and populates a target table, and where the incremental processing logic comprises a delete post processing logic configured to delete a record from the target table when:
the record has a unit of work key value in the set of unique work key values; and
the record has not been created, modified, or refreshed by execution of the program in the target table.

13. A computer-implemented method comprising:
identifying a unit of work key that is predefined for a program, where the program processes data from a data source, and further where the data source comprises one or more source tables and the unit of work key comprises one or more columns in the source tables;
accessing a unit of work driver table that records unit of work key values for records in the data source that have been created, modified, or deleted in any source table since a last time the program was executed to determine a unique set of unit of work key values;
selecting a set of records by creating a view query that selects, for the view, records from each

Assignees

• Oracle Int Corp

Inventors

• Rees John K
• Chaturvedi Rohit

Classifications

• G06F17/30418

  Physics · mapped topic

• G06F17/30336 Primary

  Physics · mapped topic

• G06F17/30368

  Physics · mapped topic

• G06F19/363

  Physics · mapped topic

• G06F16/2358 Primary

  Change logging, detection, and notification (replication G06F16/27) · CPC title