Data intake and query system search functionality in a data fabric service system

Patent metadata

Field	Value
Publication number	US-11023539-B2
Application number	US-201916264430-A
Country	US
Kind code	B2
Filing date	Jan 31, 2019
Priority date	Sep 26, 2016
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

Disclosed is a technique that can be performed in a distributed computer network. The technique can include a data index and query system that receives a search query and defines a search scheme for applying the search query on distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a network. The data index and query system communicates at least a portion of the search scheme to a search service for application on behalf of the data intake and query system, receives from the search service a search result of the search query obtained by application of the search scheme to the distributed data storage systems, and causes the search result or data indicative thereof to be displayed on a display device.

First claim

What is claimed is:

1. A method comprising:
receiving, by a data intake and query system, a search query;
defining, by the data intake and query system, a search scheme for applying the search query on a plurality of distributed data storage systems, the plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a network;
communicating, by the data intake and query system, instructions including at least a portion of the search scheme to a search service for application of the at least a portion of the search scheme on behalf of the data intake and query system, wherein, based on the instructions, the search service communicates with the internal data storage system to receive first partial results and communicates with the external data storage system to receive second partial results;
receiving, by the data intake and query system from the search service, a search result of the search query, wherein the search service generates the search result based on the first partial results and the second partial results; and
causing the search result or data indicative of the search result to be displayed on a display device.

2. The method of claim 1 , wherein the internal data storage system stores data as a plurality of time-indexed events, each of which includes a segment of raw machine data and an associated timestamp.

3. The method of claim 1 , wherein the search result is obtained by application of the portion of the search scheme to the internal data storage system.

4. The method of claim 1 , wherein the search result is obtained by application of the search scheme to the internal data storage system and the external data storage system.

5. The method of claim 1 , wherein only a portion of the search scheme is communicated to the search service.

6. The method of claim 1 , wherein a device distinct from, and communicatively coupled to, the data intake and query system implements the search service over the network.

7. The method of claim 1 , wherein the data intake and query system executes the search service.

8. The method of claim 1 , wherein the search query is input by a user and expressed in a pipelined language.

9. The method of claim 1 , wherein the search query is input by a user and expressed in a structured language.

10. The method of claim 1 , wherein the search result is rendered by the display device in a timeline visualization.

11. The method of claim 1 , wherein the search result is based on structured data retrieved from the external data storage system.

12. The method of claim 1 , wherein the search result is based on unstructured data retrieved from the internal data storage system.

13. The method of claim 1 , wherein the search result obtained by the data intake and query system is in a format specified by the search scheme.

14. The method of claim 1 , wherein the second partial results are based on structured data retrieved from the external data storage system and the first partial results are based unstructured data retrieved from the internal data storage system, and wherein the search result has a common format.

15. The method of claim 1 , wherein only a portion of the search scheme is communicated to the search service, and the search query includes a parameter causing the data intake and query system to define the portion of the search scheme that is transferred to the search service.

16. The method of claim 1 , wherein the search query includes a parameter input by a user to cause the data intake and query system to define the search scheme.

17. The method of claim 1 , wherein the at least a portion of the search scheme communicated to the search service includes a plurality of phases for obtaining data for the search result from the plurality of distributed data storage systems.

18. The method of claim 1 , wherein the at least a portion of the search scheme communicated to the search service includes a plurality of ordered phases for obtaining data in accordance with an order of the plurality of ordered phases.

19. The method of claim 1 , wherein defining the search scheme comprises:
determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems; and
defining, by the data intake and query system, a phase for searching the plurality of distributed data storage systems to retrieve data for the search result, wherein the at least a portion of the search scheme transferred to the search service includes the phase.

20. The method of claim 1 , wherein defining the search scheme comprises:
determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems; and
defining, by the data intake and query system, a plurality of phases for searching the plurality of distributed data storage systems to retrieve data for the search result, wherein the at least a portion of the search scheme transferred to the search service includes the plurality of phases.

21. The method of claim 1 , wherein the at least a portion of the search scheme includes at least one of a plurality of ordered phases for searching the external data storage system after execution of another one of the plurality of ordered phases for searching the internal data storage system.

22. The method of claim 1 , wherein defining the search scheme comprises:
determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems; and
defining a plurality of ordered phases for searching the plurality of distributed data storage systems and obtaining the search result from the search service in accordance with an order of the plurality of ordered phases;
wherein the search scheme includes the plurality of ordered phases, and the at least a portion of the search scheme includes at least one of the plurality of ordered phases for searching the external data storage system after execution of another one of the plurality of ordered phases for searching the internal data storage system.

23. The method of claim 1 further comprising, prior to receiving the search result by the data intake and query system:
determining, by the data intake and query system, that the search query requires searching the external data storage system; and
initiating, by the data intake and query system, a communications protocol with the search service to search the external data storage system.

24. The method of claim 1 further comprising, prior to receiving the search result by the data intake and query system:
determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems; and
responsive to determining that the search query requires searching the plurality of distributed data storage systems, initiating a communications search protocol with the search service and causing the communicating the instructions to the search service by the data intake and query system.

25. The method of claim 1 further comprising, prior to receiving the search result:
executing, by the data intake and query system, a first operation on the internal data storage system in accordance with the search scheme; and
sending, by

Assignees

• Splunk Inc

Inventors

• Pal Sourav
• Pride Christopher
• Bhattacharjee Arindam
• Wang Xiaowei
• Hodge James Alasdair Robert
• Ahamed Mustafa

Classifications

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/9038

  Presentation of query results · CPC title

• G06F16/211

  Schema design and management · CPC title

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title

• G06F16/248

  Presentation of query results · CPC title