Dynamically changing members of a consensus group in a distributed self-healing coordination service

Patent metadata

Field	Value
Publication number	US-2016019125-A1
Application number	US-201414334162-A
Country	US
Kind code	A1
Filing date	Jul 17, 2014
Priority date	Jul 17, 2014
Publication date	Jan 21, 2016
Grant date	—

Abstract

Systems, methods, and computer program products for managing a consensus group in a distributed computing cluster, by determining that an instance of an authority module executing on a first node, of a consensus group of nodes in the distributed computing cluster, has failed; and adding, by an instance of the authority module on a second node of the consensus group, a new node to the consensus group to replace the first node. The new node is a node in the computing cluster that was not a member of the consensus group at the time the instance of the authority module executing on the first node is determined to have failed.

First claim

What is claimed is:

1 . A method for managing a consensus group in a distributed computing cluster, the method comprising:
determining that an instance of an authority module executing on a first node, of a consensus group of nodes in the distributed computing cluster, has failed; and adding, by an instance of the authority module on a second node of the consensus group, a new node to the consensus group to replace the first node, wherein the new node comprises a node in the computing cluster that was not a member of the consensus group at the time the instance of the authority module executing on the first node is determined to have failed.

2 . The method of claim 1 , further comprising:
transmitting to the new node:
a set of nodes in the distributed computing cluster eligible to join the consensus group; and
a state of a set of locks issued to nodes in the distributed computing cluster.

3 . The method of claim 1 , wherein the instance of the authority module executing on the second node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node

4 . The method of claim 1 , further comprising:
wherein the instance of the authority module executing on the first node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node, and wherein the method further comprises, prior to adding the new node to the consensus group, electing, by remaining members of the consensus group, the instance of the authority module executing on the second node to serve as the master instance of the authority module for the consensus group.

5 . The method of claim 4 , further comprising:
transmitting, by the instance of the authority module on the second node, to each node in the distributed computing cluster, an indication that the instance of the authority module on the second node has been elected as the master authority module for the consensus group.

6 . The method of claim 1 , further comprising:
discovering a new node added to the computing cluster; adding the new node to a set of available nodes in the computing cluster; and sharing the set of available nodes with members of the consensus group.

7 . The method of claim 1 , further comprising:
removing, by the second node, the first node from the consensus group.

8 . A distributed computing cluster, comprising:
a plurality of nodes configured to provide a service to clients, each node having a processor and a memory; a consensus group of nodes formed from a subset of the plurality of nodes, the consensus group of nodes, each executing an instance of an authority module which performs operations for managing the consensus group, the operation, comprising:
determining that an instance of an authority module executing on a first node of the consensus group, has failed, and
adding, by an instance of the authority module on a second node of the consensus group, a new node to the consensus group to replace the first node, wherein the new node comprises a node in the computing cluster that was not a member of the consensus group at the time the instance of the authority module executing on the first node is determined to have failed.

9 . The distributed computing cluster of claim 8 , wherein the operation further comprises:
transmitting to the new node:
a set of nodes in the distributed computing cluster eligible to join the consensus group; and
a state of a set of locks issued to nodes in the distributed computing cluster.

10 . The distributed computing cluster of claim 8 , wherein the instance of the authority module executing on the second node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node

11 . The distributed computing cluster of claim 8 , wherein the operation further comprises:
wherein the instance of the authority module executing on the first node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node, and wherein the method further comprises, prior to adding the new node to the consensus group, electing, by remaining members of the consensus group, the instance of the authority module executing on the second node to serve as the master instance of the authority module for the consensus group.

12 . The distributed computing cluster of claim 11 , wherein the operation further comprises:
transmitting, by the instance of the authority module on the second node, to each node in the distributed computing cluster, an indication that the instance of the authority module on the second node has been elected as the master authority module for the consensus group.

13 . The distributed computing cluster of claim 8 , wherein the operation further comprises:
discovering a new node added to the computing cluster; adding the new node to a set of available nodes in the computing cluster; and sharing the set of available nodes with members of the consensus group.

14 . The distributed computing cluster of claim 8 , wherein the operation further comprises:
removing, by the second node, the first node from the consensus group.

15 . A computer-readable storage medium storing instructions, which, when executed on a processor, perform operations for managing a consensus group in a distributed computing cluster, the operation comprising:
determining that an instance of an authority module executing on a first node, of a consensus group of nodes in the distributed computing cluster, has failed; and adding, by an instance of the authority module on a second node of the consensus group, a new node to the consensus group to replace the first node, wherein the new node comprises a node in the computing cluster that was not a member of the consensus group at the time the instance of the authority module executing on the first node is determined to have failed.

16 . The computer-readable storage medium of claim 15 , further comprising:
transmitting to the new node:
a set of nodes in the distributed computing cluster eligible to join the consensus group; and
a state of a set of locks issued to nodes in the distributed computing cluster.

17 . The computer-readable storage medium of claim 15 , wherein the instance of the authority module executing on the second node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node

18 . The computer-readable storage medium of claim 15 , further comprising:
wherein the instance of the authority module executing on the first node was elected by members of the consensus group to serve as a master instance of the authority module for the consensus group prior to the failure of the instance of the authority module executing on the first node, and wherein the method further comprises, prior to adding the new node to the consensus group, electing, by remaining members of the consensus group, the instance of the authority module executing on the second node to serve as th

Assignees

• Cohesity Inc

Inventors

• Madduri Sashikanth
• Aron Mohit
• Reddy Vinay
• Gupta Vipin

Classifications

• H04L41/5096

  wherein the managed service relates to distributed or central networked applications · CPC title

• H04L69/40

  for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection (management of faults, events, alarms or notifications in data switching networks H04L41/06) · CPC title

• H04L41/12

  Discovery or management of network topologies · CPC title

• G06F11/1425

  by reconfiguration of node membership · CPC title

• H04L41/5009

  Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF] · CPC title