Real-time transactionally consistent change notifications

Patent metadata

Field	Value
Publication number	US-11354296-B2
Application number	US-201916686411-A
Country	US
Kind code	B2
Filing date	Nov 18, 2019
Priority date	May 25, 2016
Publication date	Jun 7, 2022
Grant date	Jun 7, 2022

Abstract

A method includes executing an initial instance of a change log process for a distributed system, each instance of the change log process configured to store, on memory hardware in communication with the data processing hardware, a transaction history of transactions executed on the distributed system. The method also includes receiving transaction requests for executing corresponding transactions on the distributed system and determining a change log load based on the received transaction requests. The method includes executing at least one subsequent instance of the change log process when the change log load satisfied a threshold load. When multiple instances of the change log process are executing, the method includes ceasing execution of the at least one subsequent instance of the change log process and merging the transaction history of the initial instance of the change log process and the transaction history of the at least one subsequent instance of the change log process.

First claim

What is claimed is:

1. A method comprising:
receiving, at data processing hardware, a request from a user to monitor data events for data within a distributed storage system;
identifying, by the data processing hardware, that the request seeks to receive a notification for a respective data event occurring at the distributed storage system;
determining, by the data processing hardware, a timing of the respective data event coinciding with a time when a transaction load on the distributed storage system satisfies a degradation threshold, the degradation threshold indicating a state where the transaction load degrades caching performance;
based on the timing of the respective data event, generating, by the data processing hardware, a new event log file to store a record of the respective data event rather than storing the record of the respective data event in a preexisting event log file;
recording, by the data processing hardware, the respective data event in the new event log file;
consolidating, by the data processing hardware, the new event log file and the preexisting event log file to form a consolidated event history; and
communicating, by the data processing hardware, a notification to the user based on the consolidated event history.

2. The method of claim 1 , wherein the respective data event comprises a missed event.

3. The method of claim 2 , further comprising:
generating, by the data processing hardware, a snapshot for user data, the snapshot capturing a state of the user data; and
determining, by the data processing hardware, an outcome of the missed event based on the snapshot.

4. The method of claim 1 , wherein the transaction load exceeds the degradation threshold.

5. The method of claim 1 , where communicating the notification comprises communicating the notification to a plurality of users of the distributed storage system, the plurality of users comprising the user and corresponding to subscribers that request to receive a respective notification for changes to the data stored in the distributed storage system.

6. The method of claim 1 , wherein the notification comprises the consolidated event history.

7. The method of claim 1 , wherein each event log file comprises a non-durable file format.

8. The method of claim 1 , wherein the respective data event comprises a write transaction for the data.

9. The method of claim 1 , wherein communicating the notification occurs in near real-time with the respective data event.

10. A system comprising:
data processing hardware of a distributed system; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a request from a user to monitor data events for data within a distributed storage system;
identifying that the request seeks to receive a notification for a respective data event occurring at the distributed storage system;
determining a timing of the respective data event coinciding with a time when a transaction load on the distributed storage system satisfies a degradation threshold, the degradation threshold indicating a state where the transaction load degrades caching performance;
based on the timing of the respective data event, generating a new event log file to store a record of the respective data event rather than storing the record of the respective data event in a preexisting event log file;
recording the respective data event in the new event log file;
consolidating the new event log file and the preexisting event log file to form a consolidated event history; and
communicating a notification to the user based on the consolidated event history.

11. The system of claim 10 , wherein the respective data event comprises a missed event.

12. The system of claim 11 , wherein the operations further comprise:
generating, by the data processing hardware, a snapshot for user data, the snapshot capturing a state of the user data; and
determining, by the data processing hardware, an outcome of the missed event based on the snapshot.

13. The system of claim 10 , wherein the transaction load exceeds the degradation threshold.

14. The system of claim 10 , where communicating the notification comprises communicating the notification to a plurality of users of the distributed storage system, the plurality of users comprising the user and corresponding to subscribers that request to receive a respective notification for changes to the data stored in the distributed storage system.

15. The system of claim 10 , wherein the notification comprises the consolidated event history.

16. The system of claim 10 , wherein each event log file comprises a non-durable file format.

17. The system of claim 10 , wherein the respective data event comprises a write transaction for the data.

18. The system of claim 10 , wherein communicating the notification occurs in near real-time with the respective data event.

Assignees

• Google Llc

Inventors

• Fuller Alfred
• Kumar Vijay
• Hessmer Rainer

Classifications

• G06F16/27

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/2358 Primary

  Change logging, detection, and notification (replication G06F16/27) · CPC title

• G06F9/466 Primary

  Transaction processing · CPC title

• G06F9/465

  Distributed object oriented systems (remote method invocation [RMI] G06F9/548) · CPC title

• G06F16/2379

  Updates performed during online database operations; commit processing · CPC title