System and method of replicating data in a distributed system

Patent metadata

Field	Value
Publication number	US-10650024-B2
Application number	US-201514813181-A
Country	US
Kind code	B2
Filing date	Jul 30, 2015
Priority date	Jul 30, 2015
Publication date	May 12, 2020
Grant date	May 12, 2020

Abstract

A method includes identifying a first table including data. The first table has associated metadata, an associated replication state, an associated replication log file including replication logs logging mutations of the first table, and an associated replication configuration file including a first association that associates the first table with a replication family. The method includes inserting a second association in the replication configuration file that associates a second table having a non-loadable state with the replication family. The association of the second table with the replication family causes persistence of any replication logs in the replication log file that correspond to any mutations of the first table during the existence of the second table. The method further includes generating a third table from the first table, the metadata associated with the first table, and the associated replication state of the first table.

First claim

What is claimed is:

1. A method comprising:
identifying, by data processing hardware, a first replica table comprising data, the first replica table having associated metadata defining the first replica table, an associated replication state, an associated replication log file comprising replication logs logging mutations of the first replica table, and an associated replication configuration file including a first association that associates the first replica table with a replication family;
generating, by the data processing hardware, a temporary table without data and configured in a non-loadable state, the non-loadable state preventing data from loading into the temporary table;
inserting, by the data processing hardware, a temporary association in the replication configuration file that associates the temporary table with the replication family and causes persistence of any replication logs in the replication log file that correspond to any mutations of the first replica table during an existence of the temporary table;
generating, by the data processing hardware, a second replica table from the first replica table, the metadata associated with the first replica table, and the associated replication state of the first replica table;
inserting, by the data processing hardware, a second association in the replication configuration file that associates the second replica table with the replication family; and
applying, by the data processing hardware, to the second replica table the mutations of the first replica table from any replication logs logged in the replication log file during the existence of the temporary table.

2. The method of claim 1 , further comprising removing, by the data processing hardware, the temporary association from the replication configuration file, the removal of the temporary association removing the persistence of any replication logs in the replication log file that correspond to any mutations of the first replica table during the existence of the temporary table.

3. The method of claim 2 , wherein applying to the second replica table the mutations of the first replica table of any replication logs logged in the replication log file during the existence of the temporary table comprises, before removing the temporary association from the replication configuration file:
identifying, as transient mutations, any mutations to the first replica table logged in the associated replication log file that occurred after generating the temporary table and inserting the temporary association in the replication configuration file and before a completion of generating the second replica table; and
applying the transient mutations to the second replica table.

4. The method of claim 1 , further comprising
deleting, by the data processing hardware, the temporary table after removing the temporary association from the replication configuration file.

5. The method of claim 1 , wherein the replication configuration file comprises replication logs for mutations to all tables associated with the replication family.

6. The method of claim 1 , further comprising applying, by the data processing hardware, to the second replica table the mutations of replication logs logged for each table in the replication log file during the existence of the temporary table.

7. The method of claim 1 , wherein the associated replication state of the first replica table comprises a logical clock indicating a most recent time when all mutations of a source replica table were applied to the first replica table, the first replica table being a replication of the source replica table.

8. The method of claim 7 , wherein generating the second replica table comprises:
copying a sorted string table file representing the first replica table;
copying a metadata file comprising the metadata associated with the first replica table;
copying a replication state file comprising the first replica table replication state; and
storing in memory hardware in communication with the data processing hardware, the copied sorted string table file, the copied metadata file, and the copied replication state file.

9. The method of claim 8 , wherein the sorted string table comprises a map from keys to values of the first replica table, the keys and values being arbitrary byte strings, the keys representing a row and a column, and the value representing data stored in a cell defined by the row and the column.

10. The method of claim 1 , wherein the non-loadable state of the temporary table prevents collecting replication logs of any replica tables associated with the replication family in the replication configuration file.

11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
identifying a first replica table comprising data, the first replica table having associated metadata defining the first replica table, an associated replication state, an associated replication log file comprising replication logs logging mutations of the first replica table, and an associated replication configuration file including a first association that associates the first replica table with a replication family;
generating a temporary table without data and configured in a non-loadable state, the non-loadable state preventing data from loading into the temporary table;
inserting a temporary association in the replication configuration file that associates the temporary table with the replication family and causes persistence of any replication logs in the replication log file that correspond to any mutations of the first replica table during an existence of the temporary table;
generating a second replica table from the first replica table, the metadata associated with the first replica table, and the associated replication state of the first replica table;
inserting a second association in the replication configuration file that associates the second replica table with the replication family; and
applying to the second replica table the mutations of the first replica table from any replication logs logged in the replication log file during the existence of the temporary table.

12. The system of claim 11 , wherein the operations further comprise removing the temporary association from the replication configuration file, the removal of the temporary association removing the persistence of any replication logs in the replication log file that correspond to any mutations of the first replica table during the existence of the temporary table.

13. The system of claim 12 , wherein applying to the second replica table the mutations of the first replica table of any replication logs logged in the replication log file during the existence of the temporary table comprises, before removing the temporary association from the replication configuration file:
identifying, as transient mutations, any mutations to the first replica table logged in the associated replication log file that occurred after generating the temporary table and inserting the temporary association in the replication configuration file and before a completion of generating the second replica table; and
applying the transient mutations to the second replica table.

14. The system of claim 11 , wherein the operations further comprise
deleting the temporary table after removing the temporary association from the replication configuration file.

15. The system of claim 11 , w

Assignees

• Google Inc
• Google Llc

Inventors

• Zhang Wangyuan
• Moore Li

Classifications

• G06F16/134

  Distributed indices · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F16/275 Primary

  Synchronous replication · CPC title

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/25

  Integrating or interfacing systems involving database management systems · CPC title