Switching data sources during rolling upgrade of a leader-based distributed service with minimal disruptions

Patent metadata

Field	Value
Publication number	US-10652020-B2
Application number	US-201715842624-A
Country	US
Kind code	B2
Filing date	Dec 14, 2017
Priority date	Dec 15, 2016
Publication date	May 12, 2020
Grant date	May 12, 2020

Abstract

The disclosure relates to a method of switching data sources during a rolling upgrade of a leader-based distributed service. In at least one implementation, the method includes establishing, at a computing device in a cluster of devices in a network, a watch for notification of completion of a rolling upgrade on all devices in the cluster; restarting the computing device on completion of the rolling upgrade; switching, at the computing device, from a first data source to a second data source once the rolling upgrade is complete on all devices in the cluster; if the computing device is a leader device in the cluster, migrating tasks from the first data source to the second data source after completion of the rolling upgrade; and if the computing device is a non-leader device in the cluster, determining if task migration to the second data source is done and, when done, sending a store object to the second data source.

First claim

What is claimed is:

1. A method comprising:
determining, by each of a plurality of computing devices of a cluster, that an upgrade is complete on all of the plurality of computing devices of the cluster;
in response to the determining, restarting each of the plurality of computing devices upon completion of the upgrade on all of the plurality of computing devices; and
upon restarting of all of the plurality of computing devices:
designating one of the plurality of computing devices as a leader device;
switching each of the plurality of computing devices from a first data source to a second data source;
migrating, by the leader device, a task in the first data source to the second data source;
in response to the determining, by a non-leader device that the migration of tasks is complete, executing, by the non-leader device of the plurality of computing devices, the task from the second data source instead of the first data source upon migration of the task to the second data source.

2. The method of claim 1 , further comprising setting a flag by the leader device to indicate that the migration of the task is complete.

3. The method of claim 2 , further comprising determining, by the non-leader device, that the migration of the task is complete based on the set flag.

4. The method of claim 2 , further comprising waiting, by the non-leader device, a predetermined period of time for the flag to be set before executing the task from the second data source.

5. The method of claim 1 , wherein the task that is migrated comprises a pending task or a recent task.

6. The method of claim 1 further comprising establishing, by each of the plurality of computing devices, a watch for determining that the upgrade is complete on all of the plurality of computing devices of the cluster.

7. The method of claim 5 , wherein each of the plurality of computing devices establishes the watch upon completing the upgrade.

8. The method of claim 1 , further comprising additional tasks, and wherein the leader device migrates the task and the additional tasks to the second data source from the first data source in a predetermined order.

9. A system comprising:
a plurality of computing devices of a cluster, each of the plurality of computing devices comprising a processor and a memory communicatively coupled to the processor, the memory storing computer-readable instructions that, when executed by the processor, cause the processor to:
determine that an upgrade is complete on all of the plurality of computing devices of the cluster;
in response to determining the upgrade is complete, restart all of the plurality of computing devices; and
upon restarting of all of the plurality of computing devices:
one of the plurality of computing devices is designated as a leader device;
switch each of the plurality of computing devices from a first data source to a second data source upon restarting;
the leader device migrates a task in the first data source to the second data source; and
a non-leader device of the plurality of computing devices executes the task from the second data source instead of the first data source upon determining migration of the task to the second data source is complete.

10. The system of claim 9 , wherein the leader device sets a flag to indicate that the migration of the task is complete.

11. The system of claim 10 , wherein the non-leader device determines that the migration of the task is complete based on the set flag.

12. The system of claim 10 , wherein the non-leader device waits for a predetermined period of time for the flag to be set before executing the task from the second data source.

13. The system of claim 9 , further comprising additional tasks, and wherein the leader device migrates the task and the additional tasks to the second data source from the first data source in a predetermined order.

14. The system of claim 9 , wherein each of the plurality of computing devices comprises computer-readable instructions to establish a watch to determine that the upgrade is complete on all of the plurality of computing devices of the cluster.

15. A non-transitory computer-readable media comprising computer-executable instructions embodied thereon that, when executed by a processor of each of a computing device of a plurality of computing devices of a cluster cause the respective processor to:
determine that an upgrade is complete on all of the plurality of computing devices of the cluster;
in response to the determining, restart all of the plurality of computing devices; and
upon restarting of all of the plurality of computing devices:
one of the plurality of computing devices is designated as a leader device;
switch each of the plurality of computing devices from a first data source to a second data source,
the leader device migrates a task in the first data source to the second data source; and
a non-leader device of the plurality of computing devices executes the task from the second data source instead of the first data source upon determining migration of the task to the second data source is complete.

16. The non-transitory computer-readable media of claim 15 , wherein the leader device sets a flag to indicate that the migration of the task is complete.

17. The non-transitory computer-readable media of claim 16 , wherein the non-leader device determines that the migration of the task is complete based on the set flag.

18. The non-transitory computer-readable media of claim 16 , wherein the non-leader device waits for a predetermined period of time for the flag to be set before executing the task from the second data source.

19. The non-transitory computer-readable media of claim 15 , further comprising additional tasks, and wherein the leader device migrates the task and the additional tasks to the second data source from the first data source in a predetermined order.

20. The non-transitory computer-readable media of claim 15 , wherein the processor further comprises computer-readable instructions to establish a watch to determine that the upgrade is complete on all of the plurality of computing devices of the cluster.

21. The non-transitory computer-readable media of claim 20 , wherein the processor establishes the watch upon completing the upgrade.

Assignees

• Nutanix Inc

Inventors

• Navasivasakthivelsamy Arun
• Smith Gregory A

Classifications

• G06F9/5005

  to service a request · CPC title

• G06F2009/4557

  Distribution of virtual machine instances; Migration and load balancing · CPC title

• G06F9/5088

  involving task migration · CPC title

• G06F8/656

  while running · CPC title

• H04L9/32 Primary

  including means for verifying the identity or authority of a user of the system {or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials} · CPC title