Partition tolerance in cluster membership management

Patent metadata

Field	Value
Publication number	US-9672115-B2
Application number	US-201414209426-A
Country	US
Kind code	B2
Filing date	Mar 13, 2014
Priority date	Aug 26, 2013
Publication date	Jun 6, 2017
Grant date	Jun 6, 2017

Abstract

Techniques are disclosed for managing a cluster of computing nodes following a division of the cluster into at least a first and second partition, where the cluster aggregates local storage resources of the nodes to provide an object store, and objects stored in the object store are divided into data components stored across the nodes. In accordance with one method, it is determined that a majority of data components comprising a first object are stored within nodes in the first partition. It is determined that a majority of data components comprising a second object are stored within nodes in the second partition. Configuration objects are permitted to be performed on the first object in the first partition while denying access to the first object from the second partition, and on the second object in the second partition while denying access to the second object from the first partition.

First claim

We claim:

1. A method comprising:
managing a cluster of computing nodes following a failure or management event that divides the cluster into at least a first partition and a second partition, wherein (i) the cluster aggregates local storage resources of each of the computing nodes to provide an object store, and (ii) each object stored in the object store represents a virtual disk associated with a corresponding virtual machine and contains data that is divided into respective data components stored across the storage resources of the respective computing nodes, wherein managing the cluster of computing nodes comprises:
determining that a majority of data components that comprise a first object stored in the object store are stored within computing nodes in the first partition;
determining that a majority of data components that comprise a second object stored in the object store are stored within computing nodes in the second partition;
in response to determining that the majority of data components that comprise the first object are stored within computing nodes in the first partition, permitting configuration operations to be performed on data components of the first object stored within computing nodes in the first partition while denying any access to the first object from the second partition; and
in response to determining that the majority of data components that comprise the second object are stored within computing nodes in the second partition, permitting configuration operations to be performed on data components of the second object stored within computing nodes in the second partition while denying any access to the second object from the first partition.

2. The method of claim 1 , further comprising:
determining that the majority of data components that comprise the first object further includes all data that comprises the first object; and
permitting I/O operations to be performed on the first object in the first partition by a requesting process running on one of the computing nodes in the first partition.

3. The method of claim 2 , wherein the requesting process is associated with a first virtual machine and the first object represents a first virtual disk for the first virtual machine.

4. The method of claim 1 , wherein configuration operations include operations that modify the capacity, availability, or TOPS of an object stored in the object store.

5. The method of claim 1 , wherein each of the computing nodes maintains a local copy of an in-memory database that tracks a current configuration of each object stored in the object store and indicates whether an object is accessible by the computing node.

6. The method of claim 5 , further comprising, storing in a log any operations performed on the first object by computing nodes in the first partition.

7. The method of claim 6 , wherein, upon a recovery from the failure event, the method further comprises:
replacing the local copies of the in-memory database in the computing nodes of the first partition with a replacement copy of the in-memory database from a leader node of the second partition; and
replaying the operations in the log so that the replacement copy of the in-memory database is updated to include operations performed from the first partition.

8. The method of claim 1 , wherein, upon a merging of the first partition and second partition, the method further comprises, reconciling data components of the first object stored within computing nodes in the first partition with any other data component of the first object stored in the second partition.

9. The method of claim 8 , wherein the reconciling step further comprises replaying updates made to the first object to a leader node of the merged cluster.

10. The method of claim 1 , wherein one or more data components that comprise the first object are stored within computing nodes of the second partition of computing nodes, and wherein denying any access to the first object from the second partition includes denying access to the data components of the first object stored within computing nodes in the second partition.

11. The method of claim 1 , comprising:
determining that one or more data components, but less than a majority of data components, that comprise a third object stored the object store are stored within computing nodes in the first partition of computing nodes; and
denying access to the data components of the third object from the first partition.

12. The method of claim 1 , wherein the majority of data components that comprise the first object are stored within a plurality of storage resources associated with computing nodes in the first partition.

13. A non-transitory computer readable storage medium storing instructions, which, when executed on a processor, performs operations comprising:
managing a cluster of computing nodes following a failure or management event that divides the cluster into at least a first partition and a second partition, wherein (i) the cluster aggregates local storage resources of each of the computing nodes to provide an object store, and (ii) each object stored in the object store represents a virtual disk associated with a corresponding virtual machine and contains data that is divided into respective data components stored across the storage resources of the respective computing nodes, wherein managing the cluster of computing nodes comprises:
determining that a majority of data components that comprise a first object stored in the object store are stored within computing nodes in the first partition;
determining that a majority of data components that comprise a second object stored in the object store are stored within computing nodes in the second partition;
in response to determining that the majority of data components that comprise the first object are stored within computing nodes in the first partition, permitting configuration operations to be performed on data components of the first object stored within computing nodes in the first partition while denying any access to the first object from the second partition; and
in response to determining that the majority of data components that comprise the second object are stored within computing nodes in the second partition, permitting configuration operations to be performed on data components of the second object stored within computing nodes in the second partition while denying any access to the second object from the first partition.

14. The computer readable storage medium of claim 13 , wherein the operation further comprises:
determining that the majority of data components that comprise the first object further includes all data that comprises the first object; and
permitting I/O operations to be performed on the first object in the first partition by a requesting process running on one of the computing nodes in the first partition.

15. The computer readable storage medium of claim 14 , wherein the requesting process is associated with a first virtual machine and the first object represents a first virtual disk for the first virtual machine.

16. The computer readable storage medium of claim 13 , wherein configuration operations include operations that modify the capacity, availability, or IOPS of an object stored in the object store.

17. The computer readable storage medium of claim 13 , wherein each of the computing nodes maintains a local copy of an in-memory database that tracks a current configuration of each object stored in the object store and indicates whether an object is accessible by the computing node.

Assignees

• Vmware Inc

Inventors

• Earl William
• Karamanolis Christos
• Smith Gregory A
• Knauft Eric
• Renauld Pascal

Classifications

• G06F16/13

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title

• G06F11/1484

  involving virtual machines · CPC title

• G06F11/202

  where processing functionality is redundant (redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089) · CPC title

• G06F11/1425 Primary

  by reconfiguration of node membership · CPC title

• G06F11/1464 Primary

  for networked environments · CPC title