Systems and methods for service replication, validation, and recovery in cloud-based systems

Patent metadata

Field	Value
Publication number	US-10379966-B2
Application number	US-201715813295-A
Country	US
Kind code	B2
Filing date	Nov 15, 2017
Priority date	Nov 15, 2017
Publication date	Aug 13, 2019
Grant date	Aug 13, 2019

Abstract

Systems and methods for replication and validation between a plurality of nodes in a cloud-based system forming a cluster include replicating a database and a software state between the plurality of nodes utilizing a replication algorithm which elects a master of the plurality of nodes and remaining nodes comprising peers, wherein the master is configured to perform the replicating; validating database sequences in the database at all of the plurality of nodes utilizing the replication algorithm to ensure sequential order of the validating between the plurality of nodes; and, responsive to differences between a peer and the master, performing a failure recovery in the database at the peer.

First claim

What is claimed is:

1. A method of replication and validation between a plurality of nodes in a cloud-based system forming a cluster, the method comprising:
replicating a database and a software state between the plurality of nodes utilizing a replication algorithm which elects a master of the plurality of nodes and remaining nodes as peers, wherein the master is configured to perform the replicating;
validating database sequences in the database at all of the plurality of nodes utilizing the replication algorithm to ensure sequential order of the validating between the plurality of nodes;
responsive to differences between a peer and the master, performing a failure recovery in the database at the peer; and
verifying the database of the peer and the master utilizing the replication algorithm without closing an application interacting with the database and honoring requests from the application to the database during the verifying, wherein the verifying is performed through a replication operation using the replication algorithm to perform a database dump which is verified offline with a tool for comparison, wherein the comparison omits one or more fields in the database between the peer and the master, wherein the one or more fields are inherently different.

2. The method of claim 1 , wherein the validating is performed after each database update.

3. The method of claim 1 , wherein the validating comprises:
causing each node to determine a checksum of all sequence values by the master;
providing the master's checksum to each node via the replication algorithm; and
comparing the master's checksum with a local checksum at each node.

4. The method of claim 1 , wherein the validating is performed through commands in the replication algorithm ensuring a sequential order of operations.

5. The method of claim 1 , wherein the failure recovery comprises obtaining a full dump from the master.

6. The method of claim 1 , wherein the replication algorithm comprises Raft.

7. A cloud-based system configured for replication and validation between a plurality of nodes in a cluster, the cloud-based system comprising:
a plurality of nodes each configured to replicate a database and a software state between one another utilizing a replication algorithm which elects a master of the plurality of nodes and remaining nodes as peers, wherein the master is configured to perform replication;
wherein each of the plurality of nodes is configured to validate database sequences utilizing the replication algorithm to ensure sequential order of the validation between the plurality of nodes, and
wherein, responsive to differences between a peer and the master in the validation, a failure recovery is performed for the database at the peer, wherein the database of the peer and the master are verified utilizing the replication algorithm without closing an application interacting with the database and while honoring requests from the application to the database during the verification, wherein the verification is performed through a replication operation using the replication algorithm to perform a database dump which is verified offline with a tool for comparison, and wherein the comparison omits one or more fields in the database between the peer and the master, wherein the one or more fields are inherently different.

8. The cloud-based system of claim 7 , wherein the validation is performed after each database update.

9. The cloud-based system of claim 7 , wherein the validation comprises:
a determination of a checksum of all sequence values at each node;
each node obtaining the master's checksum via the replication algorithm; and
each node comparing the master's checksum with a local checksum.

10. The cloud-based system of claim 7 , wherein the validation is performed through commands in the replication algorithm ensuring a sequential order of operations.

11. The cloud-based system of claim 7 , wherein the failure recovery comprises obtaining a full dump from the master.

12. The cloud-based system of claim 7 , wherein the replication algorithm comprises Raft.

13. A non-transitory computer-readable medium comprising instructions that, when executed, cause a processor to perform steps of:
replicating a database and a software state between the plurality of nodes utilizing a replication algorithm which elects a master of the plurality of nodes and remaining nodes as peers, wherein the master is configured to perform the replicating;
validating database sequences in the database at all of the plurality of nodes utilizing the replication algorithm to ensure sequential order of the validating between the plurality of nodes;
responsive to differences between a peer and the master, performing a failure recovery in the database at the peer; and
verifying the database of the peer and the master utilizing the replication algorithm without closing an application interacting with the database and honoring requests from the application to the database during the verifying, wherein the verifying is performed through a replication operation using the replication algorithm to perform a database dump which is verified offline with a tool for comparison, wherein the comparison omits one or more fields in the database between the peer and the master, wherein the one or more fields are inherently different.

14. The non-transitory computer-readable medium of claim 13 , wherein the validating is performed after each database update.

15. The non-transitory computer-readable medium of claim 13 , wherein the validating comprises:
causing each node to determine a checksum of all sequence values by the master;
providing the master's checksum to each node via the replication algorithm; and
comparing the master's checksum with a local checksum at each node.

16. The non-transitory computer-readable medium of claim 13 , wherein the validating is performed through commands in the replication algorithm ensuring a sequential order of operations.

17. The non-transitory computer-readable medium of claim 13 , wherein the failure recovery comprises obtaining a full dump from the master.

18. The non-transitory computer-readable medium of claim 13 , wherein the replication algorithm comprises Raft.

Assignees

• Zscaler Inc

Inventors

• Gangadharappa Tejus
• Chan Wai Leung
• Padmanabhan Dinoja
• Udupa Sivaprasad

Classifications

• G06F11/2097

  maintaining the standby controller/processing unit updated (initialisation or re-synchronisation thereof G06F11/1658 and subgroups) · CPC title

• G06F11/184

  where the redundant components implement processing functionality · CPC title

• G06F11/182

  based on mutual exchange of the output between redundant processing components · CPC title

• G06F11/0778

  Dumping, i.e. gathering error/state information after a fault for later diagnosis · CPC title

• G06F11/0757

  by exceeding a time limit, i.e. time-out, e.g. watchdogs · CPC title