Synchronous replication of high throughput streaming data

Patent metadata

Field	Value
Publication number	US-12423003-B2
Application number	US-202318166834-A
Country	US
Kind code	B2
Filing date	Feb 9, 2023
Priority date	Nov 13, 2020
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

A method for synchronous replication of stream data includes receiving a stream of data blocks for storage at a first storage location associated with a first geographical region and at a second storage location associated with a second geographical region. The method also includes synchronously writing the stream of data blocks to the first storage location and to the second storage location. While synchronously writing the stream of data blocks, the method includes determining an unrecoverable failure at the second storage location. The method also includes determining a failure point in the writing of the stream of data blocks that demarcates data blocks that were successfully written and not successfully written to the second storage location. The method also includes synchronously writing, starting at the failure point, the stream of data blocks to the first storage location and to a third storage location associated with a third geographical region.

First claim

What is claimed is:

1. A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
receiving, at a data block replicator, a stream of data blocks for storage at a first storage location of a distributed storage system and at a second storage location of the distributed storage system, the first storage location associated with a first geographical region and the second storage location associated with a second geographical region different than the first geographical region;
synchronously writing, from the data block replicator, the stream of data blocks to the first storage location and to the second storage location;
while synchronously writing the stream of data blocks to the first storage location and to the second storage location, determining an unrecoverable failure at the second storage location that prohibits further writing of the stream of data blocks to the second storage location;
determining a failure point in the writing of the stream of data blocks, the failure point demarcating data blocks that were successfully written to the second storage location and data blocks that were not successfully written to the second storage location;
synchronously writing, from the data block replicator and starting at the failure point, the stream of data blocks to the first storage location and to a third storage location of the distributed storage system simultaneously, the third storage location associated with a third geographical region different than the first geographical region and the second geographical region; and
based on determining the unrecoverable failure at the second storage location and after synchronously writing the stream of data blocks to the third storage location, asynchronously writing, from a beginning of the stream of data blocks to the failure point of the stream of data blocks, the data blocks to the third storage location from the data block replicator.

2. The method of claim 1 , wherein determining the unrecoverable failure at the second storage location that prohibits further writing of the stream of data blocks to the second storage location comprises:
determining a failure of the writing of the stream of data blocks to the second storage location;
in response to determining the failure of writing the stream of data blocks to the second storage location, retrying writing the stream of data blocks to the second storage location;
determining that retrying writing the stream of data blocks to the second storage location has failed; and
in response to determining that retrying writing the stream of data blocks to the second storage location has failed, determining that the failure is an unrecoverable failure.

3. The method of claim 1 , wherein determining the failure point in the writing of the stream of data blocks comprises reconciling a first replication log corresponding to the first storage location and a second replication log corresponding to the second storage location.

4. The method of claim 3 , wherein reconciling the first replication log and the second replication log comprises reconciling, based on a length of the first replication log and a length of the second replication log, the first replication log and the second replication log.

5. The method of claim 3 , wherein reconciling the first replication log and the second replication log comprises:
determining an index of the second replication log associated with the unrecoverable failure;
storing the index of the second replication log on memory hardware in communication with the data processing hardware;
finalizing the second replication log to prohibit further writes to the second storage location; and
generating a sentinel file to indicate a need for reconciliation.

6. The method of claim 1 , wherein the operations further comprise returning an indication of the unrecoverable failure at the second storage location to a client device associated with a user.

7. The method of claim 6 , where in the indication of the unrecoverable failure at the second storage location comprises one or more of:
the failure point;
a number of data blocks that were successfully written to the second storage location; or
an identification of the second storage location.

8. The method of claim 1 , wherein asynchronously writing the data blocks that were successfully written to the second storage location to the third storage location is performed after synchronously writing the stream of data blocks to the first storage location and to the third storage location.

9. The method of claim 1 , wherein asynchronously writing the data blocks that were successfully written to the second storage location to the third storage location is performed simultaneously while synchronously writing the stream of data blocks to the first storage location and to the third storage location.

10. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving, at a data block replicator, a stream of data blocks for storage at a first storage location of a distributed storage system and at a second storage location of the distributed storage system, the first storage location associated with a first geographical region and the second storage location associated with a second geographical region different than the first geographical region;
synchronously writing, from the data block replicator, the stream of data blocks to the first storage location and to the second storage location;
while synchronously writing the stream of data blocks to the first storage location and to the second storage location, determining an unrecoverable failure at the second storage location that prohibits further writing of the stream of data blocks to the second storage location;
determining a failure point in the writing of the stream of data blocks, the failure point demarcating data blocks that were successfully written to the second storage location and data blocks that were not successfully written to the second storage location;
synchronously writing, from the data block replicator and starting at the failure point, the stream of data blocks to the first storage location and to a third storage location of the distributed storage system simultaneously, the third storage location associated with a third geographical region different than the first geographical region and the second geographical region; and
based on determining the unrecoverable failure at the second storage location and after synchronously writing the stream of data blocks to the third storage location, asynchronously writing, from a beginning of the stream of data blocks to the failure point of the stream of data blocks, the data blocks to the third storage location from the data block replicator.

11. The system of claim 10 , wherein determining the unrecoverable failure at the second storage location that prohibits further writing of the stream of data blocks to the second storage location comprises:
determining a failure of the writing of the stream of data blocks to the second storage location;
in response to determining the failure of writing the stream of data blocks to the second storage location, retrying writing the stream of data blocks to the second storage location;
determining that retrying writing the stream of data blocks to the second storage location has failed; and
in response to determining that retrying writing the stream of data blocks to the secon

Assignees

• Google Llc

Inventors

• Edara Pavan
• Forbes Jonathan

Classifications

• G06F11/1446

  Point-in-time backing up or restoration of persistent data · CPC title

• G06F3/0653

  Monitoring storage devices or systems · CPC title

• G06F3/0635

  by changing the path, e.g. traffic rerouting, path reconfiguration · CPC title

• G06F11/3034

  where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097) · CPC title

• G06F11/1471

  involving logging of persistent data for recovery · CPC title