Data synchronization in a data analysis system comprising a data store and a metadata store

Patent metadata

Field	Value
Publication number	US-12554744-B2
Application number	US-202418885216-A
Country	US
Kind code	B2
Filing date	Sep 13, 2024
Priority date	Jun 13, 2024
Publication date	Feb 17, 2026
Grant date	Feb 17, 2026

Abstract

The present disclosure relates to a method. A main transaction may be received. The main transaction comprises a set of change records of the at least one source table. Each change record represents at least one data record of a source table of the at least one source table. A least one subset of change records of the set of change records may be determined. Each subset of change records is determined such that it represents a distinct source partition of the at least one source table. A sub-transaction may be determined per subset of change records of the at least one subset of change records. The main transaction may be executed by at least concurrently executing the set of sub-transactions, wherein the execution of each sub-transaction of the sub-transactions comprises concurrently adapting a data store and adapting a metadata store in accordance with the sub-transaction.

First claim

The invention claimed is:

1 . A method for data synchronization in a data analysis system, the data analysis system comprising a source database system and a target database system, the source database system comprising at least one source table, wherein the at least one source table comprises partitions, referred to as source partitions, the target database system comprising a data store and a metadata store, the data store being configured to comprise at least one target table corresponding to the at least one source table respectively, the metadata store being configured to comprise metadata descriptive of the at least one target table, the method comprising:
receiving a transaction, referred to as main transaction, the main transaction comprising a set of change records of the at least one source table, each change record representing a data record of a source table of the at least one source table, wherein the main transaction involving one or more source tables, referred to as transaction source tables, of the at least one source table, wherein the main transaction can involve more than one source table in case the at least one source table is multiple source tables, the method further comprising:
in response to receiving the main transaction, creating, in the metadata store, target table metadata associated with the one or more transaction source tables respectively, each target table metadata being descriptive of target partitions of a target table associated with the transaction source table, the metadata comprising the target table metadata;

determining at least one subset of change records of the set of change records where each subset of change records represents a distinct source partition of the at least one source table;
determining a transaction, referred to as sub-transaction, per subset of change records of the at least one subset of change records, resulting in a set of one or more sub-transactions;
for each sub-transaction of the set of sub-transactions:
creating, in the data store, if not yet created, one or more target partitions which are associated with a source partition of the sub-transaction in accordance with a partition scheme of the target database system;

executing the main transaction, wherein the executing comprises executing a sub-transaction in the set of sub-transactions concurrently to each other sub-transaction in the set, wherein the executing of each sub-transaction of the set of sub-transactions comprises:
concurrently configuring the data store; and
configuring the metadata store in accordance with the sub-transaction, wherein the configuring of the data store and the metadata store in accordance with the sub-transaction comprises storing data of the source partition of the sub-transaction in the associated one or more target partitions and configuring the target table metadata in accordance with the sub-transaction.

2 . The method of claim 1 , wherein the at least one source table is partitioned into the source partitions in accordance with a first partition scheme, the target database system being configured to partition the target table into target partitions in accordance with a second partition scheme, wherein the first partition scheme is the same or different from the second partition scheme, wherein in case the second partition scheme is the first partition scheme, the source partition is associated with a respective target partition, wherein in case the second partition scheme is different from the first partition scheme, the source partition is associated with respective one or more target partitions.

3 . The method of claim 1 , further comprising:
after performing the execution of the set of sub-transactions, creating in the data store a target table version per transaction source table; storing data descriptive of the created one or more target partitions in the respective created target table versions; and controlling the target database system to use the target table metadata of each created target table version for access to target partitions of the created target table version.

4 . The method of claim 3 , the target database system being configured to use current versions in the data store and in the metadata store for processing queries, the method further comprising:
marking each of the created target table metadata as an upcoming version and each of the created target table versions as an upcoming version, wherein the target database system is configured to use the upcoming versions as the current versions in response to a completion of the execution of the main transaction.

5 . The method of claim 1 , the executing of the sub-transaction comprising performing a commit of the sub-transaction, wherein the executing of the main transaction comprises performing a commit of the main transaction in response to determining that the commit of all sub-transactions is performed.

6 . The method of claim 1 , the main transaction representing a load request for loading one or more source partitions of the at least one source table, wherein the change records of the main transaction reference to data records of the one or more source partitions respectively.

7 . The method of claim 1 , wherein each change record of the main transaction is descriptive of a change of one or more data records of a source partition of the at least one source table.

8 . The method of claim 1 , further comprising:
in response to determining that the main transaction represents a load request for loading one or more source partitions of the at least one source table and that the first partition scheme is the same as the second partition scheme performing the execution of the sub-transaction partition-wise.

9 . The method of claim 1 , the main transaction involving one or more source tables, referred to as transaction source tables, of the at least one source table, the metadata comprising target table metadata associated with the one or more transaction source tables respectively, each target table metadata being descriptive of target partitions of a target table associated with the transaction source table, the method further comprising:
upon receiving the main transaction and before completion of the execution of the main transaction marking in the target table metadata data records of the data store which are replaced by the data records represented with the change records as deleted records.

10 . The method of claim 9 , the target table metadata comprising links to partitions which can be accessed by the target database system, the marking comprising removing from the target table metadata links to the partitions of the data store which are replaced by the data records represented with the change records.

11 . The method of claim 1 , further comprising:
in response to determining that the main transaction represents an incremental update request performing the execution of the sub-transaction selectively on records of the source partition.

12 . The method of claim 1 , wherein the determining of at least one subset of change records is performed using identifiers of the source partitions, wherein the identifiers are part of the main transaction or part of the metadata.

13 . The method of claim 1 , wherein the data analysis system being a data Lakehouse.

14 . The method of claim 1 , wherein the main transaction involving one source table of the at least one source table.

15 . The method of claim 1 , wherein the at least one source table being multiple source tables, the main transaction involving multiple source tables of the at least one source table

Assignees

• IBM

Inventors

• Beier Felix
• Dimov Vassil Radkov
• Butterstein Dennis
• Broll Björn
• Diedrich Dominik
• Gupta Saransh
• Waddington Daniel
• Dickey Clement Lambert

Classifications

• G06F16/2282

  Tablespace storage structures; Management thereof · CPC title

• G06F16/219

  Managing data history or versioning (querying versioned data G06F16/2474; querying temporal data G06F16/2477) · CPC title

• G06F16/2379 Primary

  Updates performed during online database operations; commit processing · CPC title

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/278 Primary

  Data partitioning, e.g. horizontal or vertical partitioning · CPC title