Parallel replication of data table partition

Patent metadata

Field	Value
Publication number	US-10902015-B2
Application number	US-201715410276-A
Country	US
Kind code	B2
Filing date	Jan 19, 2017
Priority date	Jan 19, 2017
Publication date	Jan 26, 2021
Grant date	Jan 26, 2021

Abstract

Several replication subscriptions are defined for a table in a database management system. The table is divided into partitions. Each replication subscription replicates transactions to a range of partitions. Subscriptions are assignable to different consistency groups. Transaction consistency is preserved at the consistency group level. A persistent delay table is created for each of several apply functions. Each apply function processes replication subscriptions for one consistency group, to replicate the table to a target table in parallel. Transactions for a given range of a partition are executed in parallel. When an apply function upon a row of the target table results in an error, the row is stored in the delay table. Application of each row in the delay table is repeatedly retried, and if successful, the row is removed from the delay table.

First claim

What is claimed is:

1. A replication method comprising:
defining a plurality of replication subscriptions for a table in a database management system, wherein:
the table is divided into a plurality of partitions,
each replication subscription replicates transactions to a range of partitions,
replication subscriptions are assignable to different consistency groups, and
transaction consistency is preserved at the consistency group level;

creating a persistent delay table for each of a plurality of apply functions, wherein each apply function processes one or more replication subscriptions for one consistency group;
executing, in parallel, the replication subscriptions to replicate the table to a target table, wherein:
the execution of the replication subscriptions uses the apply functions, and
within each replication subscription, transactions for a given range of a partition are executed in parallel;

in response to an apply function upon a row of the target table resulting in an error caused by an out-of-order replay of changes involving movement of a row between replication consistency groups or by a sequence of transactions that violates a global constraint across all partitions of the table, storing in a persistent delay table the row causing the error, wherein the row causing the error is delayed without disrupting the execution of the corresponding apply functions for other rows in the current transaction or other transactions;
at a predefined time interval, retrying application of each row in the persistent delay table; and
in response to a successful retried application of the row in the persistent delay table, removing the row from the persistent delay table.

2. The method of claim 1 , wherein executing the replication subscriptions further comprises:
prior to executing the transaction of the apply function, determining whether any row that is a target for the transaction is stored in the persistent delay table; and
in response to determining that the target of the row transaction is stored in the persistent delay table, processing the row in the persistent delay table, wherein for updates, the row in the persistent delay table reflects the latest updated values of the corresponding replication subscription.

3. The method of claim 1 , wherein a shared communication table is used to keep track the progress among all consistency group apply functions to identify conflicts caused by a target data mismatch that is introduced by external applications making changes at the target table, further comprising:
comparing a commit sequence value for the row in the persistent delay table to a minimum value of a latest transaction performed for all other consistency groups as logged in the shared communication table, wherein the commit sequence value is stored with the row in the persistent delay table; —in response to the commit sequence value being less than the minimum value of the latest transaction, suspending the retrying of the row in the persistent delay table; and
reporting the row in the persistent delay table as an exception.

4. The method of claim 1 , wherein, when the apply function performs an initial load of the target table that uses a spill queue mechanism, executing further comprises:
prior to applying each row in a spill queue table, comparing a partitioning key of a row in the target table to partitioning key limits of the replication subscription having initiated the initial load; and
in response to the partitioning key of the row in the target table being outside the partitioning key limits, deleting the row from the spill queue table in the event of a delete transaction, or storing the row in the persistent delay table in the event of an update transaction, and attempting to periodically force the transaction as an insert into the target table until the transaction is successfully applied to the target table by deleting the duplicate row from the conflicting partition.

5. A system for data replication comprising:
a processor;
a memory storing instructions executable by the processor;
a first database management system; and
a second database management system,
wherein the first and second database management systems are connected over a network in an active/active configuration and are synchronized using transaction replay technology, and wherein the processor is configured to execute the instructions stored in the memory to perform a method comprising:
defining a plurality of replication subscriptions for a table in the first database management system, wherein:
the table is divided into a plurality of partitions,
each replication subscription replicates transactions to a range of partitions,
replication subscriptions are assignable to different consistency groups, and
transaction consistency is preserved at the consistency group level;

creating a persistent delay table for each of a plurality of apply functions, wherein each apply function processes one or more replication subscriptions for one consistency group;
executing, in parallel, the replication subscriptions to replicate the table to a target table in the second database management system, wherein:
the execution of the replication subscriptions uses the apply functions, and
within each replication subscription, transactions for a given range of a partition are executed in parallel;

in response to an apply function upon a row of the target table resulting in an error caused by an out-of-order replay of changes involving movement of a row between replication consistency groups or by a sequence of transactions that violates a global constraint across all partitions of the table, storing in a persistent delay table the row causing the error, wherein the row causing the error is delayed without disrupting the execution of the corresponding apply functions for other rows in the current transaction or other transactions;
at a predefined time interval, retrying application of each row in the persistent delay table; and
in response to a successful retried application of the row in the persistent delay table, removing the row from the persistent delay table.

6. The system of claim 5 , wherein executing the replication subscriptions further comprises:
prior to executing the transaction of the apply function, determining whether any row that is a target for the transaction is stored in the persistent delay table; and
in response to determining that the target of the row transaction is stored in the persistent delay table, processing the row in the persistent delay table, wherein for updates, the row in the persistent delay table reflects the latest updated values of the corresponding replication subscription.

7. The system of claim 5 , wherein a shared communication table is used to keep track the progress among all consistency group apply functions to identify conflicts caused by a target data mismatch that is introduced by external applications making changes at the target table, further comprising:
comparing a commit sequence value for the row in the persistent delay table to a minimum value of a latest transaction performed for all other consistency groups as logged in the shared communication table, wherein the commit sequence value is stored with the row in the persistent delay table; —in response to the commit sequence value being less than the minimum value of the latest transaction, suspending the retrying of the row in the persistent delay table; and
reporting the row in the persistent delay table as an exception.

8. The system of claim 5 , wherein, when the apply function performs an initial load of the target table that uses a spill queue mechanism, executing further comprises:
prior to applying each row in a spill queue table,

Assignees

• IBM

Inventors

• Bourbonnais Serge
• D'Costa Austin F M
• Li Xiao
• Wierenga Jonathan W
• Wu Xin
• Zentgraf Christian
• Zhou Xiang W

Classifications

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/278 Primary

  Data partitioning, e.g. horizontal or vertical partitioning · CPC title