Fault-tolerant and highly available configuration of distributed services

Patent metadata

Field	Value
Publication number	US-10990446-B2
Application number	US-201916528625-A
Country	US
Kind code	B2
Filing date	Aug 1, 2019
Priority date	Oct 13, 2015
Publication date	Apr 27, 2021
Grant date	Apr 27, 2021

Abstract

Fault-tolerant and highly available configuration of distributed services including a computer-implemented method for role-based configuration discovery comprising receiving a request comprising an identifier of a role; identifying a first key, in a replica of a distributed configuration store, comprising a first value that matches the role identifier; identifying one or more other key-value pairs associated in the replica with the first key; and returning a response to an entity that sent the request comprising the value of at least one key-value pair that is specific to the role the service has. Also disclosed are techniques for log forwarding.

First claim

The invention claimed is:

1. A method for log forwarding, the method comprising:
obtaining, at a host, network endpoint information from a replica of a distributed configuration store that is stored locally at the host, the replica of the distributed configuration store updated using a consensus protocol that is configured to keep network endpoint information of multiple replicas of the distributed configuration store consistent;
wherein the network endpoint information identifies a location on a network of a first service, the first service comprising a centralized log collection and aggregation network service;
obtaining, at the host, an identifier of a second service installed on the host from the replica of the distributed configuration store, the second service being distinct from the first service;
based, at least in part, on the obtaining the identifier of the second service, obtaining, at the host, from the replica of the distributed configuration store, information indicating where, in a file system, one or more logs generated by the second service are stored;
using the information indicating where the one or more logs generated by the second service are stored, collecting, at the host, the one or more logs generated by the second service;
using the network endpoint information, sending, by the host, the one or more logs generated by the second service to the first service.

2. The method of claim 1 , wherein the network endpoint information comprises an interne protocol address associated with the first service for collecting logs.

3. The method of claim 1 , further comprising:
based, at least in part, on the obtaining the identifier of the second service, providing, to the first service, the identifier of the second service in association with the information from the one or more logs provided to the first service.

4. The method of claim 1 , further comprising:
obtaining a host identifier of the host at which the replica is stored; and
based, at least in part, on the obtaining the host identifier, obtaining, from the replica, information indicating where, in a file system, the one or more logs generated by the second service are stored.

5. The method of claim 4 , wherein the sending the information from the one or more logs to the first service is based, at least in part, on the obtaining the information indicating where, in the file system, the one or more logs are stored.

6. One or more non-transitory computer-readable media storing one or more programs for execution by one or more processors, the one or more programs comprising instructions for:
obtaining, at a host, network endpoint information from a replica of a distributed configuration store that is stored locally at the host, the replica of the distributed configuration store updated using a consensus protocol that is configured to keep network endpoint information of multiple replicas of the distributed configuration store consistent;
wherein the network endpoint information identifies a location on a network of a first service, the first service comprising a centralized log collection and aggregation network service;
obtaining, at the host, an identifier of a second service installed on the host from the replica of the distributed configuration store, the second service being distinct from the first service;
based, at least in part, on the obtaining the identifier of the second service, obtaining, at the host, from the replica of the distributed configuration store, information indicating where, in a file system, one or more logs generated by the second service are stored;
using the information indicating where the one or more logs generated by the second service are stored, collecting, at the host, the one or more logs generated by the second service;
using the network endpoint information, sending, by the host, the one or more logs generated by the second service to the first service.

7. The one or more non-transitory computer-readable media of claim 6 , wherein the network endpoint information comprises an internet protocol address associated with the first service for collecting logs.

8. The one or more non-transitory computer-readable media of claim 6 , further comprising instructions which, when executed by the one or more processors, cause:
based, at least in part, on the obtaining the identifier of the second service, providing, to the first service, the identifier of the second service in association with the information from the one or more logs provided to the first service.

9. The one or more non-transitory computer-readable media of claim 6 , further comprising instructions which, when executed by the one or more processors, cause:
obtaining a host identifier of the host at which the replica is stored; and
based, at least in part, on the obtaining the host identifier, obtaining, from the replica, information indicating where, in a file system, the one or more logs generated by the second service are stored.

10. The one or more non-transitory computer-readable media of claim 9 , wherein the sending the information from the one or more logs to the first service is based, at least in part, on the obtaining the information indicating where, in the file system, the one or more logs are stored.

11. A system, comprising:
one or more processors;
one or more storage media storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for:
obtaining, at a host, network endpoint information from a replica of a distributed configuration store that is stored locally at the host, the replica of the distributed configuration store updated using a consensus protocol that is configured to keep network endpoint information of multiple replicas of the distributed configuration store consistent;
wherein the network endpoint information identifies a location on a network of a first service, the first service comprising a centralized log collection and aggregation network service;
obtaining, at the host, an identifier of a second service installed on the host from the replica of the distributed configuration store, the second service being distinct from the first service;
based, at least in part, on the obtaining the identifier of the second service, obtaining, at the host, from the replica of the distributed configuration store, information indicating where, in a file system, one or more logs generated by the second service are stored;
using the information indicating where the one or more logs generated by the second service are stored, collecting, at the host, the one or more logs generated by the second service;
using the network endpoint information, sending, by the host, the one or more logs generated by the second service to the first service.

12. The system of claim 11 , wherein the network endpoint information comprises an interne protocol address associated with the first service for collecting logs.

13. The system of claim 11 , further comprising instructions which, when executed by the one or more processors, cause:
based, at least in part, on the obtaining the identifier of the second service, providing, to the first service, the identifier of the second service in association with the information from the one or more logs provided to the first service.

14. The system of claim 11 , further comprising instructions which, when executed by the one or more processors, cause:
obtaining a host identifier of the host at which the replica is stored; and
based, at least in part, on the obtaining the host identifier, obtaining, from the replica, information indicating where, in a f

Assignees

• Palantir Technologies Inc

Inventors

• Dearment Greg
• Elliot Mark
• Yu Jonathon
• Paek Jon
• Chen Phillip
• Toth Brian
• Tolnay David
• Rothfus Daniel
• Moylan Bradley

Classifications

• G06F9/44505 Primary

  Configuring for program initiating, e.g. using registry, configuration files · CPC title

• H04L67/1095 Primary

  Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes · CPC title

• G06F9/5005 Primary

  to service a request · CPC title

• G06F9/4806

  Task transfer initiation or dispatching · CPC title

• G06F8/71

  Version control (security arrangements therefor G06F21/57); Configuration management · CPC title