Data replication between databases with heterogenious data platforms

Patent metadata

Field	Value
Publication number	US-10078556-B2
Application number	US-201514841154-A
Country	US
Kind code	B2
Filing date	Aug 31, 2015
Priority date	Aug 31, 2015
Publication date	Sep 18, 2018
Grant date	Sep 18, 2018

Abstract

A system and method for data replication for databases using an intermediary server, the intermediary server choosing the order in which databases are replicated, the utilities used for each of the steps in the data replication process, the timing of the replication, and/or the timing each step of the data replication process is performed.

First claim

What is claimed is:

1. A system for data replication between databases with heterogeneous data platforms comprising:
a non-transitory memory; and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:
determining available extraction utilities for extracting an incremental from a source database;
determining a first extraction utility from the available extraction utilities;
using the first extraction utility to extract the incremental from the source database and load the incremental onto a first server;
determining available loading utilities for a target database;
determining a first loading utility from the available loading utilities;
loading the incremental from the first server to the target database; and
causing the target database to update a data table in the target database with the incremental, the updating comprising repeatedly attempting to update the data table with the incremental using an unattempted application utility from a plurality of application utilities until the data table is updated or there are no more unattempted application utilities, wherein the unattempted application utility is chosen based on a cost of using the unattempted application utility.

2. The system of claim 1 , wherein the unattempted application utility is chosen from the plurality of application utilities that have not been attempted based on a preprogrammed order.

3. The system of claim 1 , further comprising selecting the target database from a plurality of databases based on a cost of using the target database.

4. The system of claim 1 , further comprising selecting the target database from a plurality of databases based on a processor usage of the target database.

5. A computer implemented method of data replication, the method comprising:
selecting a first extraction utility from a plurality of extraction utilities based on a data format used on a source database, wherein selecting the first extraction utility is based at least in part on a second extraction utility being unavailable, and wherein the second extraction utility is unavailable because there are no available slots on the source database for the second extraction utility;
extracting an incremental from the source database and loading the incremental onto a server using the first extraction utility;
selecting a first loading utility from a plurality of loading utilities based on a data format used on a target database;
loading the incremental from the server to the target database using the first loading utility; and
causing the target database to update a data table on the target database with the incremental, the updating comprising repeatedly attempting to update the data table with the incremental using an unattempted application utility from a plurality of application utilities until the data table is updated or there are no more unattempted application utilities, wherein the unattempted application utility is chosen based on a cost of using the unattempted application utility.

6. The computer implemented method of claim 5 , wherein selecting the first extraction utility is based on an efficiency of the first extraction utility.

7. The computer implemented method of claim 5 , wherein selecting the first loading utility is based at least in part on a third loading utility failing to load the incremental.

8. The computer implemented method of claim 5 , the method further comprising transforming the extracted incremental from a first data format used on the source database to a second data format used on the target database before loading the incremental on target database.

9. A system for data replication comprising:
a source database storing a first data table having first information received prior to a time point and second information received after the time point, the source database configured to create an incremental based on the second information;
a target database storing a second data table having the first information, the target database configured to update the second data table with the incremental when the incremental is loaded in the target database such that the second data table contains both the first and second information, the updating including repeatedly attempting to update the second data table with the incremental using an unattempted application utility from a plurality of application utilities until the second data table is updated or there are no more unattempted application utilities; and
a server including one or more hardware processors configured to execute instructions to cause the server to perform operations comprising:
extracting the incremental from the source database and loading the incremental onto a non-transitory memory;
selecting the target database from a plurality of databases based on a cost of using the target database; and
loading the incremental from the non-transitory memory onto the target database.

10. The system of claim 9 , wherein the source database is configured by the server to create the incremental.

11. The system of claim 9 , wherein the target database is configured by the server to update the second data table with the incremental.

12. The system of claim 9 , wherein the source database further comprises a first cache that the incremental is stored in when created.

13. The system of claim 12 , wherein the target database further comprises a second cache that the incremental is stored in when loaded onto the target database.

14. The system of claim 9 , wherein the incremental is stored on a Hadoop® cluster when extracted from the source database.

15. The system of claim 9 , wherein the operations further comprise transforming the incremental from a first data format to a second data format.

16. The system of 15 , wherein the first data format is a data format of the source database and the second data format is a Hadoop Distributed File System (HDFS) format.

17. The system of 9 , wherein the unattempted application utility is chosen from the plurality of application utilities based on a preprogrammed order.

18. The system of 9 , wherein the unattempted application utility is chosen based on a cost of using the unattempted application utility.

19. The system of 9 , wherein the cost of using the target database is based on a processor usage of the target database.

20. The system of 9 , wherein the extracting of the incremental from the source database includes selecting a first extraction utility from a plurality of extraction utilities based on a data format used on the source database.

Assignees

• Paypal Inc

Inventors

• Clare Steve
• Sanoyan Liana
• Huang Jian
• Appavu Suresh
• Das Sandip
• Kazakov Paul
• Subramaniam Prabhagaran
• Nainar Mutharasan

Classifications

• G06F2201/82

  Solving problems relating to consistency · CPC title

• G06F2201/80

  Database-specific techniques · CPC title

• G06F11/2094

  Redundant storage or storage space (G06F11/2056 takes precedence) · CPC title

• G06F11/2097

  maintaining the standby controller/processing unit updated (initialisation or re-synchronisation thereof G06F11/1658 and subgroups) · CPC title

• G06F11/1451 Primary

  by selection of backup contents · CPC title