Third vote consensus in a cluster using shared storage devices

Patent metadata

Field	Value
Publication number	US-9836366-B2
Application number	US-201514924318-A
Country	US
Kind code	B2
Filing date	Oct 27, 2015
Priority date	Oct 27, 2015
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

A third vote consensus technique enables a first node, i.e., a surviving node, of a two-node cluster to establish a quorum and continue to operate in response to failure of a second node of the cluster. Each node maintains configuration information organized as a cluster database (CDB) which may be changed according to a consensus-based protocol. Changes to the CDB are logged on a third copy file system (TCFS) stored on a local copy of TCFS (L-TCFS). A shared copy of the TCFS (i.e., S-TCFS) may be stored on shared storage devices of one or more storage arrays coupled to the nodes. The local copy of the TCFS (i.e., L-TCFS) represents a quorum vote for each node of the cluster, while the S-TCFS represents an additional “tie-breaker” vote of a consensus-based protocol. The additional vote may be obtained from the shared storage devices by the surviving node as a third vote to establish the quorum and enable the surviving node to cast two of three votes (i.e., a majority of votes) needed to continue operation of the cluster. That is, the majority of votes allows the surviving node to update the CDB with the configuration information changes so as to continue proper operation of the cluster.

First claim

What is claimed is:

1. A method comprising:
receiving a write request directed towards a logical unit (LUN), the write request having data and received at a first node of a cluster, the first node and a second node of the cluster connected to a storage array, the first node connected to a local storage device;
maintaining configuration information of the cluster on the storage array;
maintaining a copy of the configuration information on the local storage device;
in response to the second node failing, obtaining control to update the configuration information on the storage array at the first node using a consensus protocol;
establishing a first vote of the consensus protocol by ownership of the copy of the configuration information on the local storage device; and
establishing a second vote of the consensus protocol by ownership of the configuration information on the storage array.

2. The method of claim 1 further comprising accessing the configuration information on the storage array at the first and second nodes via non-exclusive small computer systems interface (SCSI) reservations.

3. The method of claim 1 further comprising logging changes to the configuration information on the local storage device.

4. The method of claim 1 further comprising establishing a majority vote of the consensus protocol and servicing the write request at the first node.

5. The method of claim 1 further comprising updating the configuration information on the storage array in order of update events from the nodes of the cluster.

6. The method of claim 1 wherein the consensus protocol is Raft.

7. The method of claim 1 further comprising detecting failure of the second node by failure to receive a heartbeat from the second node.

8. The method of claim 1 further comprising failing to establish a quorum using the consensus protocol absent a tie-breaker vote among a number of surviving nodes.

9. The method of claim 8 wherein the second vote is the tie-breaker vote.

10. A method comprising:
receiving a write request directed towards a logical unit (LUN), the write request having data and received at a first node of a cluster, the first node and a second node of the cluster connected to a storage array;
maintaining a local copy of a configuration database of the cluster at each node of the cluster;
maintaining a shared copy of the configuration database on the storage array;
in response to the first node failing, obtaining control to update the shared copy of the configuration database at the second node using a consensus protocol;
obtaining exclusive access to the storage array at the second node;
establishing a first vote of the consensus protocol by ownership of the local copy of the configuration database at the second node;
establishing a second vote of the consensus protocol by ownership of the shared copy of the configuration database at the second node;
updating the configuration database at the second node to reflect the failure of the first node; and
servicing the write request at the second node.

11. A system comprising:
a cluster having first and second nodes, each node having a memory connected to a processor via a bus;
a storage array coupled to each node of the cluster;
a local storage device coupled to each node of the cluster;
a storage I/O stack executing on the processor of each node of the cluster, the storage I/O stack configured to:
receive a write request directed towards a logical unit (LUN), the write request having data and received at the first node of the cluster;
maintain configuration information of the cluster on the storage array;
maintain a copy of the configuration information on the local storage device;
in response to the second node failing, obtain control to update the configuration information on the storage array at the first node using a consensus protocol;
establish a first vote of the consensus protocol by ownership of the copy of the configuration information on the local storage device; and
establish a second vote of the consensus protocol by ownership of the configuration information on the storage array.

12. The system of claim 11 wherein the storage I/O stack is further configured to:
access the configuration information on the storage array at the first and second nodes via non-exclusive small computer systems interface (SCSI) reservations.

13. The system of claim 11 wherein the storage I/O stack is further configured to:
log changes to the copy of the configuration information on the local storage device.

14. The system of claim 11 wherein the storage I/O stack is further configured to:
establish a majority vote of the consensus protocol and service the write request at the first node.

15. The system of claim 11 wherein the storage I/O stack is further configured to:
update the configuration information on the storage array in order of update events from the nodes of the cluster.

16. The system of claim 11 wherein the consensus protocol is Raft.

17. The system of claim 11 wherein the storage I/O stack is further configured to:
detect failure of the second node by failure to receive a heartbeat from the second node.

18. The system of claim 11 wherein the storage I/O stack is further configured to, absent a tie-breaker vote among a number of surviving nodes, fail to establish a quorum using the consensus protocol.

19. The system of claim 18 wherein the second vote is the tie-breaker vote.

20. The system of claim 11 wherein the cluster further comprises:
a plurality of high-availability (HA) groups of nodes, wherein a first HA group includes the first and second nodes.

Assignees

• Netapp Inc

Inventors

• Schatz Bob
• Muth John
• Strange Stephen H

Classifications

• G06F11/2046

  where the redundant components share persistent storage (G06F11/2043 takes precedence) · CPC title

• G06F11/1425

  by reconfiguration of node membership · CPC title

• G06F11/2033 Primary

  switching over of hardware resources · CPC title

• G06F2201/805

  Real-time · CPC title