Failure resistant distributed computing system

Patent metadata

Field	Value
Publication number	US-11126512-B2
Application number	US-201916356090-A
Country	US
Kind code	B2
Filing date	Mar 18, 2019
Priority date	Dec 31, 2014
Publication date	Sep 21, 2021
Grant date	Sep 21, 2021

Abstract

A failure resistant distributed computing system includes primary and secondary datacenters each comprising a plurality of computerized servers. A control center selects orchestrations from a predefined list and transmits the orchestrations to the datacenters. Transmitted orchestrations include less than all machine-readable actions necessary to execute the orchestrations. The datacenters execute each received orchestration by referencing a full set of actions corresponding to the received orchestration as previously stored or programmed into the computerized server and executing the referenced full set of actions. At least one of the orchestrations comprises a failover operation from the primary datacenter to the secondary datacenter. Failover shifts performance of task from a set of processing nodes of the primary datacenter to a set of processing nodes of the secondary datacenter, such tasks including managing storage accessible by one or more remote clients and running programs on behalf of remote clients.

First claim

What is claimed is:

1. A method for performing a failover operation between a primary datacenter and a standby datacenter, comprising:
detecting, via a control center, that a failover condition exists in the primary datacenter;
selecting, via the control center, an orchestration from a predefined list of orchestrations to facilitate a failover operation from the primary datacenter to the standby datacenter;
transmitting, via a messaging queue, an indication of the selected orchestration to the primary datacenter, the standby datacenter, or both;
executing the selected orchestration to perform the failover operation from the primary datacenter to the standby datacenter; and
redirecting execution of tasks from the primary datacenter to the standby datacenter.

2. The method, as set forth in claim 1 , wherein the standby datacenter is configured to mirror changes from the primary datacenter prior to redirecting the execution of tasks, and wherein the primary datacenter is configured to mirror changes from the standby datacenter after redirecting the execution of tasks.

3. The method, as set forth in claim 1 , wherein redirecting execution of tasks from the primary datacenter to the standby datacenter comprises:
stopping all processing nodes being executed by computerized servers of the primary datacenter and of the standby datacenter.

4. The method, as set forth in claim 3 , wherein the transmitting an indication of the selected orchestration further comprises:
receiving, via at least one of the processing nodes, a request for an updated, corrected, expanded, or a combination thereof, of actions to execute the selected orchestration.

5. The method, as set forth in claim 1 , wherein redirecting execution of tasks from the primary datacenter to the standby datacenter comprises:
changing one or more connection strings in application nodes to point to the standby datacenter.

6. The method, as set forth in claim 5 , wherein redirecting execution of tasks from the primary datacenter to the standby datacenter comprises:
switching one or more DNS entries to point to a load balancer of the standby datacenter.

7. The method, as set forth in claim 1 , comprising running a discovery process on the standby datacenter and updating a configuration management database based on the discovery process.

8. A non-transitory tangible computer readable medium comprising instructions that when executed by a processor cause the processor to:
detect, via a control center, that a failover condition exists in a primary datacenter;
select, via the control center, an orchestration from a predefined list of orchestrations to facilitate a failover operation from the primary datacenter to a standby datacenter;
transmit, via a messaging queue, an indication of the selected orchestration to the primary datacenter, a secondary datacenter, or both;
stop, via the control center, processing on all active nodes of the primary datacenter and of the standby datacenter, in accordance with the selected orchestration;
route, via the control center, one or more connections of processing nodes of the primary datacenter to processing nodes of the standby datacenter, in accordance with the selected orchestration; and
restart, via the control center, all processing nodes of the primary datacenter and the standby datacenter.

9. The non-transitory tangible computer readable medium, as set forth in claim 8 , wherein detecting that the failover condition exists in the primary datacenter comprises:
testing to determine whether a transfer to the primary datacenter can complete.

10. The non-transitory tangible computer readable medium, as set forth in claim 8 , wherein the instructions that when executed by the processor cause the processor to:
run, via the control center, a post validation operation.

11. The non-transitory tangible computer readable medium, as set forth in claim 8 , wherein routing connections of the processing nodes of the primary datacenter to processing nodes of the standby datacenter comprises:
changing one or more connection strings in application nodes to point to the standby datacenter.

12. The non-transitory tangible computer readable medium, as set forth in claim 11 , wherein routing connections of processing nodes of the primary datacenter to processing nodes of the standby datacenter comprises:
switching one or more DNS entries to point to a load balancer of the standby datacenter.

13. The non-transitory tangible computer readable medium, as set forth in claim 8 , wherein the instructions, when executed by the processor, cause the processor to, after restarting all nodes of the standby datacenter, run a discovery process on the standby datacenter and update a configuration management database based on the discovery process.

14. A distributed computing system comprising:
a primary datacenter having a plurality of computerized servers and a plurality of databases;
a secondary datacenter having a plurality of computerized servers and a plurality of databases; and
a control center having a data processing machine and being communicatively coupled to the primary datacenter and the secondary datacenter, wherein the data processing machine of the control center is configured to provide an instruction to perform a failover operation to at least one of the computerized servers of the primary datacenter and/or secondary datacenter to cause the failover operation from the primary datacenter to the secondary datacenter to be conducted, wherein the failover operation comprises:
detecting, via the control center, that a failover condition exists in the primary datacenter;
selecting, via the control center, an orchestration from a predefined list of orchestrations to facilitate the failover operation from the primary datacenter to the secondary datacenter;
transmitting, via a messaging queue, an indication of the selected orchestration to the primary datacenter, the secondary datacenter, or both;
stopping, via the control center, processing on the primary datacenter and the secondary datacenter;
redirecting, via the control center, communications from the primary datacenter to the secondary datacenter; and
restarting, via the control center, the primary datacenter and the secondary datacenter.

15. The system, as set forth in claim 14 , in response to the detection that the failover condition exists, wherein the control center is configured to provide the instruction to perform the failover operation.

16. The system, as set forth in claim 14 , wherein stopping processing on the primary datacenter and the secondary datacenter comprises:
stopping all processing nodes being executed by the plurality of computerized servers of the primary datacenter and by the plurality of computerized servers of the secondary datacenter.

17. The system, as set forth in claim 14 , wherein the at least one of the computerized server does not perform the instruction to perform the failover operation, and wherein the at least one computerized server delegates the instruction to a different computerized server.

18. The system, as set forth in claim 14 , wherein redirecting communications from the primary datacenter to the secondary datacenter comprises:
changing one or more connection strings in application nodes to point to the secondary datacenter as a new primary datacenter.

19. The system, as set forth in claim 18 , wherein redirecting communications from the primary datacenter to the secondary datacenter comprises:
switching one or more DNS entries to point t

Assignees

• Servicenow Inc

Inventors

• Chandrashekar Sridhar
• Patel Swapnesh
• Shah Viral
• Garg Anurag
• Chablani Anjali

Classifications

• H04L61/4511

  using domain name system [DNS] · CPC title

• G06F2201/85

  Active fault masking without idle spares · CPC title

• G06F11/2025 Primary

  using centralised failover control functionality · CPC title

• G06F11/2007

  using redundant communication media · CPC title

• G06F11/0709

  in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems · CPC title