Search service for a data fabric system

Patent metadata

Field	Value
Publication number	US-11010435-B2
Application number	US-201916264441-A
Country	US
Kind code	B2
Filing date	Jan 31, 2019
Priority date	Sep 26, 2016
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

Disclosed is a technique that can be performed in a distributed network. The technique can include a search service system that receives an indication of at least a portion of a search scheme to cause worker nodes to obtain search results from distributed data storage systems. The search scheme is defined by a data intake and query system. The search service system defines a search process based on the at least a portion of the search scheme and executes the search process to cause the worker nodes to obtain search results from the distributed data storage systems. The search service system receives a combination of search results based on the search results obtained by the worker nodes from the distributed data storage systems, and causes an output based on the combination of search results obtained by the data intake and query system in accordance with the search scheme.

First claim

What is claimed is:

1. A method comprising:
updating a dynamic list of worker nodes, wherein the dynamic list of worker nodes is updated based on a determined availability of a set of worker nodes;
receiving, by a search service, an indication of at least a portion of a search scheme to cause worker nodes to obtain search results from a plurality of distributed data storage systems, the search scheme being defined by a data intake and query system, and the plurality of distributed data storage systems including at least one of an external data storage system communicatively coupled to the data intake and query system over a network or an internal data storage system of the data intake and query system;
selecting, by the search service, a plurality of worker nodes from the dynamic list of worker nodes;
defining, by the search service, a search process based on the at least a portion of the search scheme defined by the data intake and query system;
executing, by the search service, the search process to cause the plurality of worker nodes to obtain a plurality of search results from the plurality of distributed data storage systems;
receiving, by the search service, a combination of search results based on the plurality of search results obtained by the plurality of worker nodes from the plurality of distributed data storage systems; and
causing, by the search service, an output based on the combination of search results obtained by the data intake and query system in accordance with the search scheme.

2. The method of claim 1 , wherein only a portion of the search scheme is received by the search service.

3. The method of claim 1 , wherein the plurality of search results is obtained from the external data storage system and the internal data storage system of the data intake and query system.

4. The method of claim 1 , wherein at least some of the plurality of search results are time-ordered events retrieved from the internal data storage system, each time-ordered event including a segment of raw machine data and an associated timestamp.

5. The method of claim 1 , wherein the combination of search results is derived from the plurality of search results obtained from the external data storage system and the internal data storage system of the data intake and query system.

6. The method of claim 1 , wherein the combination of search results includes a plurality of time-ordered events.

7. The method of claim 1 , wherein the combination of search results is an aggregation of the plurality of search results collected by the plurality of worker nodes.

8. The method of claim 1 , wherein the combination of search results includes structured data retrieved from the external data storage system.

9. The method of claim 1 , wherein the combination of search results includes unstructured data retrieved from the internal data storage system.

10. The method of claim 1 , wherein the combination of search results communicated to the data intake and query system is in a format specified in the search scheme.

11. The method of claim 1 , wherein the combination of search results is based on structured data retrieved from the external data storage system and unstructured data retrieved from the internal data storage system.

12. The method of claim 1 , wherein the combination of search results includes data retrieved in parallel from peer indexers of the internal data storage system.

13. The method of claim 1 , wherein a device distinct from, and communicatively coupled to, the data intake and query system implements the search service over the network.

14. The method of claim 1 , wherein the data intake and query system implements the search service.

15. The method of claim 1 , wherein causing the output based on the combination of search results comprises:
causing the combination of search results or data indicative of the combination of search results to be displayed on a display device.

16. The method of claim 1 , wherein the output is rendered on a display device in a timeline visualization.

17. The method of claim 1 , wherein the search scheme is based on user input expressed in a pipelined language.

18. The method of claim 1 , wherein the search scheme is based on user input expressed in a structured language.

19. The method of claim 1 , wherein the search scheme comprises a plurality of phases, and defining the search process comprises:
defining, by the search service, the search process as a directed acyclic graph including at least some of the plurality of phases.

20. The method of claim 1 , wherein the search scheme comprises a plurality of phases, and defining the search process comprises:
defining, by the search service, the search process to include a first phase to cause the plurality of worker nodes to obtain at least some of the plurality of search results from the external data storage system and a second phase to cause the plurality of worker nodes to obtain at least some of the plurality of search results from the internal data storage system.

21. The method of claim 1 , wherein the search scheme comprises a plurality of ordered search phases, and defining the search process comprises:
defining, by the search service, the search process to include a first phase to cause the plurality of worker nodes to obtain at least some of the plurality of search results from the external data storage system followed by a second phase to cause the plurality of worker nodes to obtain at least some of the plurality of search results from the internal data storage system.

22. The method of claim 1 , wherein the search scheme comprises a plurality of phases, and defining the search process comprises:
defining, by the search service, the search process include a first phase to cause the plurality of worker nodes to obtain at least some of the plurality of search results from the external data storage system, followed by a second phase to cause the plurality of worker nodes to obtain at least some of the plurality of search results from the internal data storage system, followed by a third phase to cause the search service to obtain the plurality of search results from the plurality of worker nodes.

23. The method of claim 1 , wherein the plurality of search results includes first search results obtained from the internal data storage system in a first format, second search results obtained from the external data storage system in a second format different from the first format, and the combination of search results is in a common format.

24. The method of claim 1 , wherein the plurality of search results includes a plurality of time-indexed events obtained from the internal data storage system, data obtained from the external data storage system in a format other than time-indexed events, and the combination of search results is in a specified format.

25. The method of claim 1 further comprising:
metering an amount of time and computing resources utilized to complete the search process.

26. The method of claim 1 , wherein at least some of the plurality of search results are retrieved from the external data storage system in accordance with a first portion of the search scheme and at least some of the plurality of search results are retrieved from the internal data storage system in accordance with a second portion of the search scheme that is different from the first portion of the search scheme.

Assignees

• Splunk Inc

Inventors

• Pal Sourav
• Pride Christopher
• Bhattacharjee Arindam
• Wang Xiaowei
• Hodge James Alasdair Robert
• Ahamed Mustafa

Classifications

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/90335

  Query processing · CPC title

• G06F16/904

  Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957) · CPC title

• G06F16/2455

  Query execution · CPC title

• G06F16/252

  between a Database Management System and a front-end application · CPC title