Parallel scanners for log based replication

Patent metadata

Field	Value
Publication number	US-9836516-B2
Application number	US-201414171390-A
Country	US
Kind code	B2
Filing date	Feb 3, 2014
Priority date	Oct 18, 2013
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

Log based replication using parallel scanners is described. An example method includes establishing replication paths from a primary database to a plurality of destination servers. A plurality of scanner threads are spawned to scan a transaction log of the primary database. Specifically, one scanner thread is assigned to each replication path corresponding to each destination server. For each scanner thread, data bound to a replication path associated with the scanner thread is selected from a subset of the transaction log. Accordingly, the data is distributed to a destination server associated with the replication path.

First claim

What is claimed is:

1. A computer-implemented method for data replication, comprising:
establishing a first replication path from a primary database to a first destination server and a second replication path from the primary database to a second destination server;
spawning a first scanner thread and a second scanner thread, wherein the first and second scanner threads scan one or more log records in a transaction log of the primary database, the first scanner thread mapped to the first replication path and the second scanner thread mapped to the second replication path, wherein the first scanner threads selects data from the transaction log for the first replication path, and the second scanner thread selects data from the transaction log for the second replication path; and
spawning a first sender thread and a second sender thread, wherein the first sender thread is mapped to the first scanner thread and the second sender thread is mapped to the second scanner thread, the first sender thread distributes the data selected by the first scanner thread to the first destination server, the second sender thread distributes the data selected by the second scanner thread to the second destination server, and the first and second sender threads distribute the selected data in parallel to the first and second destination servers.

2. The method of claim 1 , wherein the first scanner thread generates first Log Transfer Language (LTL) commands based on the data selected by the first scanner thread, and the second scanner thread generates second LTL commands based on the data selected by the second scanner thread.

3. The method of claim 1 , wherein the first scanner thread filters out log records in the transaction log based on a first data binding defined for the first replication path, and the second scanner thread filters out log records in the transaction log based on a second data binding defined for the second replication path.

4. The method of claim 1 , further comprising:
defining a priority replication path, wherein a dedicated scanner thread is mapped to the priority replication path, and the dedicated scanner thread is reserved for selecting priority data from the transaction log for the priority replication path.

5. The method of claim 1 , further comprising:
defining a default replication path, wherein a default scanner thread is mapped to the default replication path, and the default scanner thread is reserved for selecting unspecified data from the transaction log for the default replication path.

6. The method of claim 1 , further comprising:
implementing a load balancing mechanism among the first and second scanner threads based on path performance characteristics of the first and second replication paths.

7. The method of claim 6 , further comprising:
defining a fast replication path, wherein a larger number of scanner threads are assigned to the fast replication path than are assigned to a non-fast replication path.

8. The method of claim 1 , further comprising:
assigning a plurality of scanner threads to a single replication path.

9. The method of claim 1 ,
wherein the first sender thread is dedicated to distributing the data selected by the first scanner thread across the first replication path, and the second sender thread is dedicated to distributing the data selected by the second scanner thread across the second replication path.

10. The method of claim 1 , further comprising:
building a first internal object related structure, wherein the first scanner thread uses the first internal object related structure to process log records in the transaction log associated with the first replication path; and
building a second internal object related structure, wherein the second scanner thread uses the second internal object related structure to process log records in the transaction log associated with the second replication path.

11. The method of claim 1 , further comprising:
assigning a plurality of replication paths to a plurality of destination servers in a one-to-one mapping.

12. The method of claim 1 , wherein the first replication path is associated with a portion of the transaction log based on an object binding.

13. A system for memory management, comprising:
one or more processors;
a path establisher, configured to establish, using the one or more processors, a first replication path from a primary database to a first destination server and a second replication path from the primary database to a second destination server;
a scanner creator, configured to:
spawn, using the one or more processors, a first scanner thread and a second scanner thread, wherein the first and second scanner threads scan one or more log records in a transaction log of the primary database, the first scanner thread mapped to the first replication path and the second scanner thread mapped to the second replication path, wherein the first scanner threads selects data from the transaction log for the first replication path, and the second scanner thread selects data from the transaction log for the second replication path; and

a data distributor, configured to:
spawn, using the one or more processors, a first sender thread and a second sender thread, wherein the first sender thread is mapped to the first scanner thread and the second sender thread is mapped to the second scanner thread, the first sender thread distributes the data selected by the first scanner thread to the first destination server, the second sender thread distributes the data selected by the second scanner thread to the second destination server, and the first and second sender threads distribute the selected data in parallel to the first and second destination servers.

14. The system of claim 13 , further comprising:
a command generator, configured to generate, using the one or more processors, first Log Transfer Language (LTL) commands based on the data selected by the first scanner thread, and second LTL commands based on the data selected by the second scanner thread.

15. The system of claim 13 , further comprising:
a log filter, configured to filter out, using the one or more processors, first log records in the transaction log based on a first data binding defined for the first replication path, and second log records in the transaction log based on a second data binding defined for the second replication path.

16. The system of claim 13 , further comprising:
a path definer, configured to define, using the one or more processors, a priority replication path, wherein a dedicated scanner thread is mapped to the priority replication path, and the dedicated scanner thread is reserved for selecting priority data from the transaction log for the priority replication path.

17. The system of claim 13 , further comprising:
a path definer, configured to define, using the one or more processors, a default replication path, wherein a default scanner thread is mapped to the default replication path, and the default scanner thread is reserved for selecting unspecified data from the transaction log for the default replication path.

18. The system of claim 13 , further comprising:
a load balancer, configured to implement, using the one or more processors, a load balancing mechanism among the first and second scanner threads based on path performance characteristics of the first and second replication paths.

19. The system of claim 18 , further comprising:
a path definer, configured to define, using the one or more processors, a fast replication path, whe

Assignees

• Quakkelaar Rene
• Siles Rene
• Lora Elena
• Sap Se

Inventors

• Quakkelaar Rene
• Siles Rene
• Lora Elena

Classifications

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F17/30575 Primary

  Physics · mapped topic