Continuing operation of a quorum based system after failures

Patent metadata

Field	Value
Publication number	US-10049011-B2
Application number	US-201615144950-A
Country	US
Kind code	B2
Filing date	May 3, 2016
Priority date	May 3, 2016
Publication date	Aug 14, 2018
Grant date	Aug 14, 2018

Abstract

A processor-implemented method, for continuing operation of a quorum based system is provided. The method detects a loss of quorum. A plurality of speculative configurations is created, whereby each speculative configuration is isolated from other speculative configurations in the quorum based system. Each speculative configuration continues to order requests during the creation of speculative configurations. The method selects and starts one of the plurality of speculative configurations as a new operational configuration. Ordered requests continue to the new operational configuration. The original configuration of the quorum based system is restarted in response to the plurality of speculative configurations not being isolated.

First claim

What is claimed is:

1. A processor-implemented method for continuing operation of a quorum based system comprising:
detecting partitioning of the quorum based system as a result of a loss of quorum after one or more of a network failure, a software process failure, and a server hardware failure;
creating a plurality of speculative configurations of one or more functioning nodes of the quorum based system in response to the detected partitioning, wherein each speculative configuration is isolated from other speculative configurations in the quorum based system, and wherein each speculative configuration continues to order requests while the plurality of speculative configurations is created;
selecting and starting one of the plurality of speculative configurations as a new operational configuration, wherein the ordered requests are added to the new operational configuration; and
restarting an original configuration of the quorum based system, in response to the plurality of speculative configurations not being isolated, wherein restarting the original configuration comprises:
restarting a number of processes, wherein the number of processes is a minimum quorum of the original configuration;
synchronizing state information among the restarted number of processes, wherein the state information includes notifying the restarted number of processes that the original configuration is no longer active;
altering quorum definitions of the restarted number of processes and the processes in the new operational configuration to the original configuration; and
restarting and integrating any remaining failed processes into the original configuration.

2. The method of claim 1 , wherein the starting further comprises:
selecting the new operational configuration from the plurality of speculative configurations;
stopping all processes belonging to the original configuration of the quorum based system but not listed in the new operational configuration; and
altering quorum definitions of all of the processes belonging to the original configuration listed in the new operational configuration, wherein all of the processes belonging to the original configuration are removed from the original configuration and added to the new operational configuration.

3. The method of claim 1 , wherein the configurations of the plurality of speculative configurations are automatically created based on a plurality of partitioning rules.

4. The method of claim 3 , wherein based on one of the plurality of partitioning rules, the speculative configuration having a most processes is the new operational configuration.

5. The method of claim 3 , wherein based on one of the plurality of partitioning rules, the new operational configuration is the speculative configuration having a highest priority.

6. The method of claim 3 , wherein the new operational configuration is selected based on the plurality of partitioning rules, or is manually selected from a display of available speculative configurations.

7. A computer program product for continuing operation of a quorum based system comprising:
a plurality of agents embodied on a computer readable storage medium, the plurality of agents including program instructions executable by a processor, the program instructions comprising:
program instructions to detect partitioning of the quorum based system as a result of a loss of quorum after one or more of a network failure, a software process failure, and a server hardware failure;
program instructions to create a plurality of speculative configurations of one or more functioning nodes of the quorum based system in response to the detected partitioning, wherein each speculative configuration is isolated from other speculative configurations in the quorum based system, and wherein each speculative configuration continues to order requests while the plurality of speculative configurations is created;
program instructions to select and start one of the plurality of speculative configurations as a new operational configuration, wherein the ordered requests are added to the new operational configuration; and
program instructions to restart an original configuration of the quorum based system, in response to the plurality of speculative configurations not being isolated, wherein the program instructions to restart the original configuration comprise:
program instructions to restart a number of processes, wherein the number of processes is a minimum quorum of the original configuration;
program instructions to synchronize state information among the restarted number of processes, wherein the state information includes notifying the restarted number of processes that the original configuration is no longer active;
program instructions to alter quorum definitions of the restarted number of processes and the processes in the new operational configuration to the original configuration; and
program instructions to restart and integrating any remaining failed processes into the original configuration.

8. The computer program product of claim 7 , wherein the program instructions to start further comprise:
program instructions to select the new operational configuration from the plurality of speculative configurations;
program instructions to stop all processes belonging to the original configuration of the quorum based system but not listed in the new operational configuration; and
program instructions to alter quorum definitions of all of the processes belonging to the original configuration listed in the new operational configuration, wherein all of the processes belonging to the original configuration are removed from the original configuration and added to the new operational configuration.

9. The computer program product of claim 7 , wherein the configurations of the plurality of speculative configurations are automatically created based on a plurality of partitioning rules.

10. The computer program product of claim 9 , wherein based on one of the plurality of partitioning rules, the speculative configuration having a most processes is the new operational configuration.

11. The computer program product of claim 9 , wherein based on one of the plurality of partitioning rules, the new operational configuration is the speculative configuration having a highest priority.

12. A computer system for continuing operation of a quorum based system comprising:
one or more processors, one or more computer-readable storage devices, and a plurality of program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors, the plurality of program instructions comprising:
program instructions to detect partitioning of the quorum based system as a result of a loss of quorum after one or more of a network failure, a software process failure, and a server hardware failure;
program instructions to create a plurality of speculative configurations of one or more functioning nodes of the quorum based system in response to the detected partitioning, wherein each speculative configuration is isolated from other speculative configurations in the quorum based system, and wherein each speculative configuration continues to order requests while the plurality of speculative configurations is created;
program instructions to select and start one of the plurality of speculative configurations as a new operational configuration, wherein the ordered requests continue to the new operational configuration; and
program instructions to restart an original configuration of the quorum based system, in response to the plurality of speculative configurations not being isolated, wherein

Assignees

• IBM

Inventors

• Bortnikov Vita
• Cahana Zvi
• Shachor Shlomit I
• Shnayderman Ilya

Classifications

• G06F11/181

  Eliminating the failing redundant component · CPC title

• G06F11/182 Primary

  based on mutual exchange of the output between redundant processing components · CPC title

• G06F2201/85

  Active fault masking without idle spares · CPC title

• G06F11/142 Primary

  Reconfiguring to eliminate the error (group management mechanisms in a peer-to-peer network H04L67/1044) · CPC title

• G06F2201/805

  Real-time · CPC title