Search functionality of a data intake and query system

Patent metadata

Field	Value
Publication number	US-11176208-B2
Application number	US-201916570545-A
Country	US
Kind code	B2
Filing date	Sep 13, 2019
Priority date	Sep 26, 2016
Publication date	Nov 16, 2021
Grant date	Nov 16, 2021

Abstract

Disclosed is a technique that can be performed in a distributed computer network. The technique can include a data index and query system that receives a search query, defines a search scheme for applying the search query on distributed data storage systems including an internal data storage system of the data index and query system and an external data storage system. The internal data storage system stores data as time-indexed events including respective segments of raw machine data. The data index and query system can transfer a portion of the search scheme to a search service, which can return search results obtained by application of the search scheme to the distributed data storage systems including the internal data storage system and the external data storage system. Lastly, the search results or data indicative of the search results can be output on a display device to the user.

First claim

What is claimed is:

1. A method comprising:
defining, by a data intake and query system, a search scheme for applying a search query on a plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a computer network;
transferring, by the data intake and query system, a first portion of the search scheme to a search service;
receiving, by the data intake and query system from the search service, a first search result of the search query obtained by application of the first portion of the search scheme to at least the external data storage system of the plurality of distributed data storage systems;
producing, by the data intake and query system, a second search result of the search query by applying a second portion of the search scheme to the internal data storage system; and
causing, by the data intake and query system, the first and second search results or an output indicative of the first and second search results to be displayed by a display device.

2. The method of claim 1 , wherein the internal data storage system stores data as a plurality of time-indexed events, each of which includes a segment of raw machine data and an associated timestamp.

3. The method of claim 1 , further comprising:
transmitting, by the data intake and query system, the first search result to a worker node communicatively coupled to the external data storage system over the computer network.

4. The method of claim 1 , wherein the search service is implemented by a device that is communicatively coupled to the data intake and query system over the computer network.

5. The method of claim 1 , wherein the internal data storage system stores data in data stores of indexers that are accessible to the data intake and query system and the search service.

6. The method of claim 1 , further comprising:
receiving, by the data intake and query system, input indicative of the search query that is expressed in a pipelined language.

7. The method of claim 1 , further comprising:
receiving, by the data intake and query system, input indicative of the search query that is expressed in a structured language.

8. The method of claim 1 , wherein the output is rendered by the display device in a timeline visualization.

9. The method of claim 1 ,
wherein the search query includes a parameter that causes the data intake and query system to define the first portion of the search scheme to be transferred to the search service, and
wherein the parameter is specified by a user responsible for submitting the search query.

10. The method of claim 1 , wherein defining the search scheme comprises:
determining that the search query requires searching the plurality of distributed data storage systems; and
defining a plurality of phases based on the search query, the plurality of phases including:
a first phase for searching the plurality of distributed data storage systems, and
a second phase for processing search results obtained in the first phase,

wherein the first portion of the search scheme includes at least one of the plurality of phases.

11. The method of claim 1 , wherein defining the search scheme comprises:
determining that the search query requires searching the plurality of distributed data storage systems; and
defining a plurality of phases based on the search query, the plurality of phases including:
a first phase for searching the plurality of distributed data storage systems, and
a second phase for processing search results obtained in the first phase,

wherein the first portion of the search scheme includes at least one of the plurality of phases, and
wherein the first phase is performed by a plurality of indexers executing in parallel.

12. The method of claim 1 , wherein defining the search scheme comprises:
determining that the search query requires searching the plurality of distributed data storage systems; and
defining a plurality of ordered phases based on the search query, the plurality of ordered phases including:
a first phase for searching the plurality of distributed data storage systems, and
a second phase for processing search results obtained in the first phase,

wherein the first portion of the search scheme includes at least one of the plurality of ordered phases.

13. The method of claim 1 , further comprising:
executing, by the data intake and query system, a first search operation on the internal data storage system in accordance with the search scheme; and
sending, by the data intake and query system to the search service, a message that causes the search service to initiate a second search operation for searching the external data storage system in accordance with the search scheme.

14. The method of claim 1 , wherein the first search result is based on structured data stored in the external data storage system.

15. The method of claim 1 , wherein the second search result is based on unstructured data stored in the internal data storage system.

16. The method of claim 1 , wherein the first search result, the second search result, or both are in a format specified by the search scheme.

17. The method of claim 1 ,
wherein the first search result is based on structured data stored in the external data storage system,
wherein the second search result is based on unstructured data stored in the internal data storage system, and
wherein the method further comprises:
transforming, by the data intake and query system, the first and second search results into a common format to facilitate aggregation of partial search results obtained in response to the search query.

18. The method of claim 1 ,
wherein the first search result is based on structured data stored in the external data storage system,
wherein the second search result is based on unstructured data stored in the internal data storage system, and
wherein the method further comprises:
transforming, by the data intake and query system, the first and second search results into a common format to facilitate aggregation of partial search results obtained in response to the search query, wherein the first and second search results are transformed into data events having structures that are compatible with the data intake and query system.

19. The method of claim 1 , wherein the search query includes a parameter that causes the data intake and query system to define the first portion of the search scheme.

20. The method of claim 1 , wherein the search query includes a parameter that cause the data intake and query system to define the first portion of the search scheme, and wherein the parameter is specified by a user responsible for submitting the search query.

21. The method of claim 1 , wherein the first portion of the search scheme transferred to the search service includes a plurality of phases for obtaining data from the plurality of distributed data storage systems.

22. The method of claim 1 , wherein the first portion of the search scheme transferred to the search service includes a plurality of ordered phases for obtaining data from the plurality of distributed data storage systems in accordance with an order of the plurality of ordered phases.

23. The method of claim 1 , wherein the first search result of the search query is obtained by application of the first portion of the search scheme to th

Assignees

• Splunk Inc

Inventors

• Pal Sourav
• Pride Christopher
• Bhattacharjee Arindam
• Wang Xiaowei
• Hodge James Alasdair Robert
• Ahamed Mustafa

Classifications

• G06F16/27 Primary

  Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor · CPC title

• G06F16/252

  between a Database Management System and a front-end application · CPC title

• G06F16/212

  with details for data modelling support · CPC title

• G06F16/248

  Presentation of query results · CPC title

• G06F16/90335

  Query processing · CPC title