Replicating large statements with low latency

Patent metadata

Field	Value
Publication number	US-12056153-B2
Application number	US-202217674914-A
Country	US
Kind code	B2
Filing date	Feb 18, 2022
Priority date	Dec 3, 2018
Publication date	Aug 6, 2024
Grant date	Aug 6, 2024

Abstract

A method, system and computer program product is provided. The method includes identifying a large statement, writing rows that are changed by the large statement into a set of files outside a database management system recovery log, and writing log records for the set of files to the database management system recovery log, such that the log records provide ordering of row changes made by the large statement. The log records include a log sequence number corresponding to a first row changed and a format and encoding used for the row changes. The method further includes streaming, by the one or more processors of the computer system, the set of files out-of-band by a replication capture process to a target database management system.

First claim

What is claimed is:

1. A computer-implemented method comprising:
identifying, by one or more processors of a source database management computer system at a first site, a large statement when the large statement is a single statement that is larger than a threshold of at least 10 megabytes in size;
writing, by the one or more processors of the source database management computer system at the first site, rows that are changed by the large statement into a set of files outside a database management system recovery log;
writing, by the one or more processors of the source database management computer system at the first site, log records for the set of files to the database management system recovery log, such that the log records provide ordering of row changes made by the large statement, wherein the log records include a log sequence number corresponding to a first row changed and a format and encoding used for the row changes;
streaming, by the one or more processors of the source database management computer system at the first site, the set of files out-of-band by a replication capture process to a target database management system at a second site located a physical disaster recovery distance apart from the first site; and
transmitting, by the one or more processors of the source database management computer system at the first site, transactions through an inline transmission interface, wherein the transactions include coordination messages from the source database management computer system at the first site to the target database management system at a second site, wherein the coordinated messages are associated with the set of files streamed out-of-band for preserving commit order, wherein the coordination messages include log sequence numbers (LSNs) for preserving the commit order.

2. The method of claim 1 , further comprising:
optimizing, by the one or more processors of the source database management computer system at the first site, the set of files in a format that provides for fast replay by an apply process at the target database management system.

3. The method of claim 1 , wherein the identifying further includes:
determining, by the one or more processors of the source database management computer system at the first site, that the large statement is larger than a configurable threshold.

4. The method of claim 1 , wherein the writing the log records further comprises:
generating, by the one or more processors of the source database management computer system at the first site, multiple written log records, each containing a subset of log data pertaining to the large statement.

5. The method of claim 1 , further comprising:
transmitting, by the one or more processors of the source database management computer system at the first site, the large statement to the target database management system at the second site and applying the large statement at the target database management system at the second site in parallel to execution at the source database management computer system at the first site without impacting the parallelism of applying other transactions between the source database management computer system at the first site and the target database management system at the second site; and
achieving a replication latency of less than a few seconds while preserving transaction integrity and ensuring that dependent transactions are applied in the correct order.

6. The method of claim 1 , further comprising:
receiving, by the one or more processors of the source database management computer system at the first site, a notification from the target database management system at the second site via a file transfer control queue when files from the set of files are applied; and
deleting, by the one or more processors of the source database management computer system at the first site, files from the set of files after receiving the notification.

7. The method of claim 1 , further comprising:
separately grouping, by the one or more processors of the source database management computer system at the first site, the written set of files based on the type of operation.

8. A computer system, comprising:
one or more processors of a source database management computer system at a first site;
one or more memory devices coupled to the one or more processors of the source database management computer system at the first site; and
one or more computer readable storage devices coupled to the one or more processors of the source database management computer system at the first site, wherein the one or more storage devices contain program code executable by the one or more processors of the source database management computer system at the first site via the one or more memory devices to implement a method of replication, the method comprising:
identifying, by the one or more processors of the source database management computer system at a first site, a large statement when the large statement is a single statement that is larger than a threshold of at least 10 megabytes in size;
writing, by the one or more processors of the source database management computer system at the first site, rows that are changed by the large statement into a set of files outside a database management system recovery log;
writing, by the one or more processors of the source database management computer system at the first site, log records for the set of files to the database management system recovery log, such that the log records provide ordering of row changes made by the large statement, wherein the log records include a log sequence number corresponding to a first row changed and a format and encoding used for the row changes;
streaming, by the one or more processors of the source database management computer system at the first site, the set of files out-of-band by a replication capture process to a target database management system at a second site located a physical disaster recovery distance apart from the first site;
transmitting, by the one or more processors of the source database management computer system at the first site, transactions through an inline transmission interface, wherein the transactions include coordination messages from the source database management computer system at the first site to the target database management system at a second site, wherein the coordinated messages are associated with the set of files streamed out-of-band for preserving commit order, wherein the coordination messages include log sequence numbers (LSNs) for preserving the commit order.

9. The computer system of claim 8 , the method further comprising:
optimizing, by the one or more processors of the source database management computer system at the first site, the set of files in a format that provides for fast replay by an apply process at the target database management system.

10. The computer system of claim 8 , wherein the identifying further includes:
determining, by the one or more processors of the source database management computer system at the first site, that the large statement is larger than a configurable threshold.

11. The computer system of claim 8 , wherein the writing the log records further comprises:
generating, by the one or more processors of the source database management computer system at the first site, multiple written log records, each containing a subset of log data pertaining to the large statement.

12. The computer system of claim 8 , the method further comprising:
transmitting, by the one or more processors of the source database management computer system at the first site, the large statement to the target database management

Assignees

• IBM

Inventors

• Bourbonnais Serge
• D'Costa Austin
• Zheng Roger
• Zentgraf Christian
• Mooney David

Classifications

• G06F16/24532

  of parallel queries · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title

• G06F16/2379 Primary

  Updates performed during online database operations; commit processing · CPC title

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title