Fault-tolerant and highly-available configuration of distributed services

Patent metadata

Field	Value
Publication number	US-9910697-B2
Application number	US-201615284957-A
Country	US
Kind code	B2
Filing date	Oct 4, 2016
Priority date	Oct 13, 2015
Publication date	Mar 6, 2018
Grant date	Mar 6, 2018

Abstract

Fault-tolerant and highly available configuration of distributed services including a computer-implemented method for distributed task orchestration. The method includes setting a watch on a first key in a first replica of a distributed configuration store and setting a value of a second key in a second replica of the distributed configuration store where the value of the second key indicates that a first sub-task of a distributed task is successfully complete. The value of the second key is received based on the watch set on the first key and execution of the second sub-tasks is initiated based on receiving the value of the second key. Also disclosed are techniques for service lifecycle management.

First claim

The invention claimed is:

1. A method for distributed task orchestration, the method comprising:
setting a watch on a first key in a first replica of a distributed configuration store;
setting a value of a second key in a second replica of the distributed configuration store, the value of the second key indicating that a first sub-task of a distributed task is successfully complete;
wherein the second replica comprises the first key and the second key;
wherein the value of the second key set in the second replica is replicated from the second replica to the first replica based at least in part on a consensus protocol;
wherein the first replica comprises the first key and the second key;
wherein the first replica and the second replica are stored at different hosts;
based, at least in part, on the watch set on the first key, receiving the value of the second key from the first replica; and
based, at least in part, on the receiving the value of the second key from the first replica, initiating execution of a second sub-task of the distributed task that is dependent on successful completion of the first sub-task.

2. The method of claim 1 , wherein the watch set of the first key is a forward watch.

3. The method of claim 1 , wherein:
the first replica is stored at a first host;
the second replica is stored at a second host that is not the first host;
the setting the watch on the first key, the receiving the value of the second key, and the initiating the execution of the second sub-task are performed at the first host; and
the setting the value of the second key is performed at the second host.

4. The method of claim 1 , wherein the setting the value of the second key is based, at least in part, on successfully completing execution of the first sub-task.

5. The method of claim 1 , further comprising:
based, at least in part, on task dependency information stored in the first replica, determining that initiating execution of the second sub-task is dependent on successful completion of the first sub-task.

6. One or more non-transitory computer-readable media storing one or more programs for execution by one or more processors, the one or more programs comprising instructions for:
setting a watch on a first key in a first replica of a distributed configuration store;
setting a value of a second key in a second replica of the distributed configuration store, the value of the second key indicating that a first sub-task of a distributed task is successfully complete;
wherein the second replica comprises the first key and the second key;
wherein the value of the second key set in the second replica is replicated from the second replica to the first replica based at least in part on a consensus protocol;
wherein the first replica comprises the first key and the second key;
wherein the first replica and the second replica are stored at different hosts;
based, at least in part, on the watch set on the first key, receiving the value of the second key from the first replica; and
based, at least in part, on the receiving the value of the second key from the first replica, initiating execution of a second sub-task of the distributed task that is dependent on successful completion of the first sub-task.

7. The one or more non-transitory computer-readable media of claim 6 , wherein the watch set of the first key is a forward watch.

8. The one or more non-transitory computer-readable media of claim 6 , wherein:
the first replica is stored at a first host;
the second replica is stored at a second host that is not the first host;
the setting the watch on the first key, the receiving the value of the second key, and the initiating the execution of the second sub-task are performed at the first host; and
the setting the value of the second key is performed at the second host.

9. The one or more non-transitory computer-readable media of claim 6 , wherein the setting the value of the second key is based, at least in part, on successfully completing execution of the first sub-task.

10. The one or more non-transitory computer-readable media of claim 6 , the one or more programs further comprising instructions for:
based, at least in part, on task dependency information stored in the first replica, determining that initiating execution of the second sub-task is dependent on successful completion of the first sub-task.

11. A system, comprising:
one or more processors;
one or more storage media storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for:
setting a watch on a first key in a first replica of a distributed configuration store;
setting a value of a second key in a second replica of the distributed configuration store, the value of the second key indicating that a first sub-task of a distributed task is successfully complete;
wherein the second replica comprises the first key and the second key;
wherein the value of the second key set in the second replica is replicated from the second replica to the first replica based at least in part on a consensus protocol;
wherein the first replica comprises the first key and the second key;
wherein the first replica and the second replica are stored at different hosts;
based, at least in part, on the watch set on the first key, receiving the value of the second key from the first replica; and
based, at least in part, on the receiving the value of the second key from the first replica, initiating execution of a second sub-task of the distributed task that is dependent on successful completion of the first sub-task.

12. The system of claim 11 , wherein the watch set of the first key is a forward watch.

13. The system of claim 11 , wherein:
the first replica is stored at a first host;
the second replica is stored at a second host that is not the first host;
the setting the watch on the first key, the receiving the value of the second key, and the initiating the execution of the second sub-task are performed at the first host; and
the setting the value of the second key is performed at the second host.

14. The system of claim 11 , wherein the setting the value of the second key is based, at least in part, on successfully completing execution of the first sub-task.

15. The system of claim 11 , the one or more programs further comprising instructions for:
based, at least in part, on task dependency information stored in the first replica, determining that initiating execution of the second sub-task is dependent on successful completion of the first sub-task.

16. The system of claim 11 , the one or more programs further comprising instructions for:
setting a watch on a third key in the second replica;
based, at least in part, on successful completion of the second sub-task, setting a value of a fourth key in the first replica;
based, at least in part, on the watch set on the third key, receiving the value of the fourth key; and
based, at least in part, on the receiving the value of the fourth key, initiating execution of a third sub-task of the distributed task that is dependent on successful completion of the second sub-task.

17. The system of claim 11 , wherein:
the first replica is stored at a first host;
the second replica is stored at a second host that is not the first host;
the one or more programs further comprise instructions for:
performing the setting the watch on the first key in the first replica at the first host;
performing the setting the value of the second key in the second replica at the second host;
p

Assignees

• Palantir Technologies Inc
• Palantir Technologies Inc

Inventors

• Dearment Greg

Classifications

• G06F8/71

  Version control (security arrangements therefor G06F21/57); Configuration management · CPC title

• G06F9/44

  Arrangements for executing specific programs · CPC title

• G06F9/4843

  by program, e.g. task dispatcher, supervisor, operating system · CPC title

• G06F9/465

  Distributed object oriented systems (remote method invocation [RMI] G06F9/548) · CPC title

• G06F9/50

  Allocation of resources, e.g. of the central processing unit [CPU] · CPC title