Replication of summary data in a clustered computing environment

Patent metadata

Field	Value
Publication number	US-10387448-B2
Application number	US-201514929089-A
Country	US
Kind code	B2
Filing date	Oct 30, 2015
Priority date	May 15, 2012
Publication date	Aug 20, 2019
Grant date	Aug 20, 2019

Abstract

Techniques and mechanisms are disclosed to increase the availability of summary data within a clustered data intake and query system by replicating the summary data within the cluster. In general, summary data may store “pre-computed” results for one or more search queries and can be used by indexers of a cluster to process subsequent instances of the same search queries. At a high level, replication of summary data within a cluster may include ensuring that each instance of summary data created by an indexer of a cluster is replicated to other indexers within the cluster that store copies of the same grouped subset(s) of data to which the summary data relates. In this manner, if one or more indexers of an indexer cluster fail, other indexers of the cluster can make immediate use of replicated copies of the summary data without re-creating it.

First claim

What is claimed is:

1. A method comprising:
determining, at a master node of a group of indexers, that an indexer of the group of indexers has summary data for a grouped subset of data stored by the indexer, the grouped subset of data including raw data received by the indexer and representative of the performance or operation of one or more components of an information technology (IT) environment, the summary data storing one or more pre-computed results based on at least one search query for data from the grouped subset of data, wherein subsequent instances of the at least one search query retrieve the one or more pre-computed results from the summary data, wherein the group of indexers operate as a cluster and wherein the master node coordinates replication and searching of data across the group of indexers in the cluster;
identifying, by the master node, one or more other indexers of the group of indexers that store a copy of the grouped subset of data and do not have a copy of the summary data, wherein the master node designates which indexers in the cluster have responsibility for responding to queries associated with the grouped subset of data;
causing the summary data to be retrieved from the indexer and sent to the identified one or more other indexers of the group of indexers.

2. The method of claim 1 , wherein the summary data is associated with a report, and wherein the at least one search query is a recurring search used to generate the report.

3. The method of claim 1 , wherein the summary data is associated with a data model for data stored by the group of indexers, and wherein the one or more pre-computed results correspond to fields of the data model.

4. The method of claim 1 , wherein the summary data is searchable by queries other than the at least one search query.

5. The method of claim 1 , wherein the one or more pre-computed results are based on a transformation of data returned from the grouped subset of data based on the least one search query, the transformation including one or more of: an aggregation, a filtering, and a data conversion.

6. The method of claim 1 , wherein determining that an indexer of a group of indexers has summary data for a grouped subset of data stored by the indexer includes receiving, by the master node of the group of indexers, a message from the indexer indicating that the indexer has the summary data.

7. The method of claim 1 , further comprising sending, by the master node of the group of indexers, summary data replication instructions to the indexer, the summary data replication instructions causing the indexer to send the summary data to the identified one or more other indexers of the group of indexers.

8. The method of claim 1 , wherein the indexer cluster is configured to replicate grouped subsets of data within the indexer cluster based on a data replication policy.

9. The method of claim 1 , wherein determining that an indexer of a group of indexers has summary data for a grouped subset of data stored by the indexer is based on performing, by the master node of the group of indexers, a scheduled search for new summary data stored by the indexer.

10. The method of claim 1 , further comprising:
receiving, from each indexer of the group of indexers, summary metadata indicating a set of summary data instances stored by the indexer, each summary data instance associated with a particular grouped subset of data stored by the indexer;
generating, based on the summary metadata, a summary data manifest indicating, for each indexer of the group of indexers, summary data instances stored by the indexer;
wherein identifying the one or more other indexers of the group of indexers includes determining that the summary data manifest does not include the summary data.

11. The method of claim 1 , further comprising:
receiving, from each indexer of the group of indexers, summary metadata indicating a set of summary data instances stored by the indexer, each summary data instance associated with a particular grouped subset of data stored by the indexer;
generating, based on the summary metadata, a summary data manifest indicating, for each indexer of the group of indexers, summary data instances stored by the indexer;
in response to determining that the indexer has the summary data for the grouped subset of data stored by the indexer, updating the summary data manifest to include the summary data.

12. The method of claim 1 , wherein the indexer stores the summary data separate from the grouped subset of data.

13. The method of claim 1 , wherein the grouped subset of data includes the summary data.

14. The method of claim 1 , further comprising:
receiving, by the master node of the group of indexers, a copy of the summary data from the indexer;
wherein causing the summary data to be sent to the identified one or more other indexers of the group of indexers includes sending, by the master node, the copy of the summary data to the one or more other indexers.

15. The method of claim 1 , wherein causing the summary data to be sent to the identified one or more other indexers of the group of indexers includes sending, by the master node of the group of indexers, summary data replication instructions to the indexer, the summary data replication instructions causing the indexer to send the summary data to the identified one or more other indexers of the group of indexers.

16. The method of claim 1 , wherein causing the summary data to be sent to the identified one or more other indexers of the group of indexers includes sending, by the master node of the group of indexers, summary data replication instructions to the indexer, the summary data replication instructions causing the indexer to send the summary data to the identified one or more other indexers of the group of indexers; and
wherein the summary data replication instructions include separate commands to send the summary data to each indexer of the identified one or more other indexers.

17. The method of claim 1 , wherein determining that the indexer has summary data includes determining that the indexer has new summary data for the grouped subset of data.

18. The method of claim 1 , wherein determining that the indexer has summary data includes determining that the indexer has updated summary data for the grouped subset of data.

19. The method of claim 1 , wherein the summary data is associated with an identifier of a search head that sent the specified search query.

20. The method of claim 1 , further comprising:
receiving, from the indexer of the group of indexers, a message indicating that the indexer has deleted existing summary data for a grouped subset of data stored by the indexer;
sending instructions to the identified one or more other indexers to delete the existing summary data stored at the identified one or more other indexers.

21. The method of claim 1 , further comprising:
receiving, from the indexer of the group of indexers, a message indicating that the indexer has trimmed existing summary data for a grouped subset of data stored by the indexer;
sending instructions to the identified one or more other indexers to trim the existing summary data stored at the identified one or more peer indexers.

22. The method of claim 1 , wherein the indexer has primary responsibility for responding to search queries for data stored by the grouped subset of data, and wherein the one or more other indexers do not respond to searches for data stored by the copy of the grou

Assignees

• Splunk Inc

Inventors

• Xu Da
• Vasan Sundar
• Bhagi Dhruva Kumar

Classifications

• G06F2201/86

  Event-based monitoring · CPC title

• G06F2201/875

  Monitoring of systems including the internet · CPC title

• G06F11/2094

  Redundant storage or storage space (G06F11/2056 takes precedence) · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/065

  Replication mechanisms · CPC title