Hybrid Cluster-Based Data Intake and Query

Patent metadata

Field	Value
Publication number	US-2016092558-A1
Application number	US-201414526493-A
Country	US
Kind code	A1
Filing date	Oct 28, 2014
Priority date	Sep 30, 2014
Publication date	Mar 31, 2016
Grant date	—

Abstract

Various embodiments describe multi-site cluster-based data intake and query systems, including cloud-based data intake and query systems. Using a hybrid search system that includes cloud-based data intake and query systems working in concert with so-called “on-premises” data intake and query systems can promote the scalability of search functionality. In addition, the hybrid search system can enable data isolation in a manner in which sensitive data is maintained “on premises” and information or data that is not sensitive can be moved to the cloud-based system. Further, the cloud-based system can enable efficient leveraging of data that may already exist in the cloud.

First claim

What is claimed is:

1 . A method comprising:
receiving, at a first cluster, a search query; contacting one or more clusters other than the first cluster to request information on their active indexers, at least one of the one or more clusters comprising a cloud-based cluster and said contacting occurring through a firewall of either the first cluster or the cloud-based cluster; obtaining, from the cloud-based cluster, the information associated with active indexers of the cloud-based cluster; distributing the search query to active indexers of the cloud-based cluster and the first cluster, said distributing to the active indexers of the cloud-based cluster occurring through the firewall of the cloud-based cluster; and receiving a response from individual indexers of the cloud-based cluster that includes event results associated with the search query.

2 . A method as described in claim 1 , wherein said contacting occurs through the firewall of the cloud-based cluster.

3 . A method as described in claim 1 , wherein said contacting occurs through the firewall of the first cluster.

4 . A method as described in claim 1 , wherein the search query is configured to search events associated with timestamps, the events comprising raw portions of machine data.

5 . A method as described in claim 1 , wherein the search query is configured to be used in connection with a late-binding schema.

6 . A method as described in claim 1 , wherein the search query is configured to be used to search log data.

7 . A method as described in claim 1 , wherein the search query is configured to be used to wire data.

8 . A method as described in claim 1 , wherein said first cluster is an on-premises cluster.

9 . A method as described in claim 1 , wherein said first cluster and said cloud-based cluster include respective master nodes that are knowledgeable about active indexers within their respective cluster.

10 . A method as described in claim 1 , further comprising:
receiving, at the cloud-based cluster, the request for information on active indexers; preparing a response including information on how to communicate with the active indexers; and sending the response to a cluster from which the request was received.

11 . A method as described in claim 10 , wherein the firewall of the cloud-based cluster is configured to allow inbound communication based on an IP address of a search head that requests said information.

12 . A method as described in claim 10 , wherein the firewall of the cloud-based cluster is configured to allow inbound communication based on an IP address associated with a cluster that performs said contacting.

13 . A method as described in claim 1 , wherein the information associated with active indexers of the cloud-based cluster includes respective IP addresses of the active indexers.

14 . A method as described in claim 1 , wherein the information associated with active indexers of the cloud-based cluster includes respective IP addresses of the active indexers and a generation identifier to be used in distributing the search query.

15 . A method as described in claim 1 , wherein said obtaining is performed by obtaining said information from a master node of the cloud-based cluster, the master node being configured to return a list of active indexers and a generation identifier that is to be used in distributing the search query, generation identifiers identifying primary and secondary indexers of the cloud-based cluster.

16 . A method as described in claim 1 , wherein the first cluster maintains information required to be maintained with a higher level of security than data in the cloud-based cluster.

17 . A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors, perform operations comprising:
receiving, at a first cluster, a search query; contacting one or more clusters other than the first cluster to request information on their active indexers, at least one of the one or more clusters comprising a cloud-based cluster and said contacting occurring through a firewall of either the first cluster or the cloud-based cluster; obtaining, from the cloud-based cluster, the information associated with active indexers of the cloud-based cluster; distributing the search query to active indexers of the cloud-based cluster and the first cluster, said distributing to the active indexers of the cloud-based cluster occurring through the firewall of the cloud-based cluster; and receiving a response from individual indexers of the cloud-based cluster that includes event results associated with the search query.

18 . A non-transitory computer readable storage medium as described in claim 17 , wherein said contacting occurs through the firewall of the cloud-based cluster.

19 . A non-transitory computer readable storage medium as described in claim 17 , wherein said contacting occurs through the firewall of the first cluster.

20 . A non-transitory computer readable storage medium as described in claim 17 , wherein the search query is configured to search events associated with timestamps, the events comprising raw portions of machine data.

21 . A non-transitory computer readable storage medium as described in claim 17 , wherein the search query is configured to be used in connection with a late-binding schema.

22 . A non-transitory computer readable storage medium as described in claim 17 , wherein the search query is configured to be used to search log data.

23 . A non-transitory computer readable storage medium as described in claim 17 , wherein the search query is configured to be used to wire data.

24 . A non-transitory computer readable storage medium as described in claim 17 , wherein said first cluster is an on-premises cluster.

25 . A non-transitory computer readable storage medium as described in claim 17 , wherein said first cluster and said cloud-based cluster include respective master nodes that are knowledgeable of active indexers within their respective cluster for a particular search.

26 . A non-transitory computer readable storage medium as described in claim 17 , wherein the information associated with active indexers of the cloud-based cluster includes respective IP addresses of the active indexers.

27 . A non-transitory computer readable storage medium as described in claim 17 , wherein the information associated with active indexers of the cloud-based cluster includes respective IP addresses of the active indexers and a generation identifier to be used in distributing the search query.

28 . A non-transitory computer readable storage medium as described in claim 17 , wherein said obtaining is performed by obtaining said information from a master node of the cloud-based cluster, the master node being configured to return a list of active indexers and a generation identifier that is to be used in distributing the search query, generation identifiers identifying primary and secondary indexers of the cloud-based cluster.

29 . A non-transitory computer readable storage medium as described in claim 17 , wherein the first cluster is configured to maintain sensitive information, and the cloud-based cluster is configured to maintain information that is not sensitive information.

30 . A system comprising

Assignees

• Splunk Inc

Inventors

• Ago Ledio
• Shanaghy Declan Gerard

Classifications

• G06F16/328

  Management therefor · CPC title

• G06F16/35 Primary

  Clustering; Classification · CPC title

• G06F17/30705 Primary

  Physics · mapped topic

• G06F17/30631

  Physics · mapped topic