System and method for capture of change data from distributed data sources, for use with heterogeneous targets

Patent metadata

Field	Value
Publication number	US-2019102418-A1
Application number	US-201816145707-A
Country	US
Kind code	A1
Filing date	Sep 28, 2018
Priority date	Sep 29, 2017
Publication date	Apr 4, 2019
Grant date	—

Abstract

In accordance with an embodiment, described herein is a system and method for capture of change data from a distributed data source system, for example a distributed database or a distributed data stream, and preparation of a canonical format output, for use with one or more heterogeneous targets, for example a database or message queue. The change data capture system can include support for features such as distributed source topology-awareness, initial load, deduplication, and recovery. A technical purpose of the systems and methods described herein includes determination and communication of changes performed to data at a distributed data source that includes a large amount of data across a plurality of nodes, to one or more target computer systems.

First claim

What is claimed is:

1 . A system for capture of change data from a distributed data source, for use with heterogeneous targets, comprising:
a computer that includes a processor, and a change data capture process manager executing thereon; wherein the change data capture process manager is configured to capture change data from a distributed data source, using a capture process, for use with one or more targets.

2 . The system of claim 1 , wherein the distributed data source is one of a distributed database, or a distributed data stream, or other distributed data source, and wherein the one or more targets include one or more of a database, message queue, or other target.

3 . The system of claim 1 , wherein the change data capture process manager performs a change data capture process that converts the change data read from the distributed data source, into a canonical format output of the change data, for consumption by the one or more targets.

4 . The system of claim 3 , whereupon based on a target system to which the change data will be communicated, the canonical format output of the change data is converted to a format used by the target system.

5 . The system of claim 3 , wherein the change data capture process manager enables support for a new target system to be provided by a pluggable adapter component that reads the canonical format output of the change data and converts it to a format used by the new target system.

6 . The system of claim 1 , wherein the change data capture process manager performs a deduplication process that provides automatic deduplication of the data provided by the distributed data source.

7 . The system of claim 1 , wherein the change data capture process manager performs automatic discovery of a distributed source topology associated with the distributed data source system, and provides access to one or more distributed source change trace entity at nodes of the distributed data source system.

8 . The system of claim 6 , whereupon a change to the distributed source topology associated with the distributed data source system, including one or more nodes being added to or removed from the distributed source topology, the deduplication process detects the change to the distributed source topology.

9 . The system of claim 1 , whereupon the change data capture process manager determining that a particular node in the distributed data source system, which had been providing records, becomes unavailable, the change data capture process manager performs a recovery process that selects a replica node at which to obtain records.

10 . The system of claim 1 , wherein if there is more than one replica node with a matching last record, a replica with the maximum record history is selected to feed a partition token found in the last record processed by the unavailable node.

11 . A method for capture of change data from a distributed data source, for use with heterogeneous targets, comprising:
providing, at a computer, a change data capture process manager executing thereon; capturing change data from a distributed data source, using a capture process, for use with one or more targets.

12 . The method of claim 11 , wherein the distributed data source is one of a distributed database, or a distributed data stream, or other distributed data source, and wherein the one or more targets include one or more of a database, message queue, or other target.

13 . The method of claim 11 , wherein the change data capture process manager performs a change data capture process that converts the change data read from the distributed data source, into a canonical format output of the change data, for consumption by the one or more targets.

14 . The method of claim 13 , whereupon based on a target system to which the change data will be communicated, the canonical format output of the change data is converted to a format used by the target system.

15 . The method of claim 13 , wherein the change data capture process manager enables support for a new target system to be provided by a pluggable adapter component that reads the canonical format output of the change data and converts it to a format used by the new target system.

16 . The method of claim 11 , wherein the change data capture process manager performs a deduplication process that provides automatic deduplication of the data provided by the distributed data source.

17 . The method of claim 11 , wherein the change data capture process manager performs automatic discovery of a distributed source topology associated with the distributed data source system, and provides access to one or more distributed source change trace entity at nodes of the distributed data source system.

18 . The method of claim 16 , whereupon a change to the distributed source topology associated with the distributed data source system, including one or more nodes being added to or removed from the distributed source topology, the deduplication process detects the change to the distributed source topology.

19 . The method of claim 11 , whereupon the change data capture process manager determining that a particular node in the distributed data source system, which had been providing records, becomes unavailable, the change data capture process manager performs a recovery process that selects a replica node at which to obtain records.

20 . The method of claim 11 , wherein if there is more than one replica node with a matching last record, a replica with the maximum record history is selected to feed a partition token found in the last record processed by the unavailable node.

21 . A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:
providing, at a computer, a change data capture process manager executing thereon; capturing change data from a distributed data source, using a capture process, for use with one or more targets.

Assignees

• Oracle Int Corp

Inventors

• Vasudevan Sanal
• Haryanto Lego
• Corbin Scott Roger

Classifications

• G06F16/2358 Primary

  Change logging, detection, and notification (replication G06F16/27) · CPC title

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F17/30368 Primary

  Physics · mapped topic

• G06F17/30371

  Physics · mapped topic

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title