Multi-phased execution of a search query

Patent metadata

Field	Value
Publication number	US-11625404-B2
Application number	US-201916687158-A
Country	US
Kind code	B2
Filing date	Nov 18, 2019
Priority date	Jan 30, 2017
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

The disclosed embodiments include a method performed by a data intake and query system. The method includes receiving a search query by a search head, defining a search process for applying the search query to indexers, delegating a first portion of the search process to indexers and a second portion of the search process to intermediary node(s) communicatively coupled to the search head and the indexers. The first portion can define a search scope for obtaining partial search results of the indexers and the second portion can define operations for combining the partial search results by the intermediary node(s) to produce a combination of the partial search results. The search head then receives the combination of the partial search results, and outputs final search results for the search query, where the final search results are based on the combination of the partial search results.

First claim

What is claimed is:

1. A method comprising:
receiving, by a data intake and query system, a search query;
defining, by the data intake and query system, a search process based on the search query;
delegating, by the data intake and query system to a plurality of indexers, a first portion of the search process that defines a search scope for obtaining partial search results by the plurality of indexers;
delegating, by the data intake and query system to an intermediary node, a second portion of the search process that defines operations for combining the partial search results obtained by the plurality of indexers to produce a combination of partial search results;
receiving, by the data intake and query system, the combination of partial search results; and
outputting, by the data intake and query system, final search results for the search query that are based on the combination of partial search results;
wherein the intermediary node is one of a plurality of intermediary nodes to which the second portion of the search process is delegated, and
wherein the second portion of the search process defines operations for combining the partial search results by the plurality of intermediary nodes to produce combinations of partial search results including the combination of partial search results.

2. The method of claim 1 ,
wherein the second portion of the search process defines operations for combining the partial search results in parallel across the plurality of intermediary nodes to produce combinations of partial search results including the combination of partial search results.

3. The method of claim 1 , wherein the intermediary node is a physical server that produces the combination of partial search results.

4. The method of claim 1 , wherein the intermediary node is a logical node that produces the combination of partial search results.

5. The method of claim 1 ,
wherein the intermediary node is a physical server,
wherein said delegating of the second portion of the search process comprises delegating the second portion of the search process to a plurality of physical servers including the physical server, and
wherein the second portion of the search process defines operations for combining the partial search results by the plurality of physical servers to produce combinations of partial search results including the combination of partial search results.

6. The method of claim 1 ,
wherein the intermediary node is a logical node,
wherein said delegating of the second portion of the search process comprises delegating the second portion of the search process to a plurality of logical nodes including the logical node, and
wherein the second portion of the search process defines operations for combining the partial search results by the plurality of logical nodes to produce combinations of partial search results including the combination of partial search results.

7. The method of claim 1 , wherein the search process comprises a plurality of directed acyclic executable phases to be executed by the plurality of indexers and the intermediary node.

8. The method of claim 1 , wherein said defining comprises defining the search process based on a plurality of policies.

9. The method of claim 1 ,
wherein said defining comprises defining a plurality of execution phases based on a plurality of policies,
wherein the first portion of the search process is a first subset of the plurality of execution phases, and
wherein the second portion of the search process is a second subset of the plurality of execution phases.

10. The method of claim 1 ,
wherein said defining comprises defining a plurality of ordered execution phases based on a plurality of policies,
wherein the first portion of the search process is a first subset of the plurality of ordered execution phases, and
wherein the second portion of the search process is a second subset of the plurality of ordered execution phases.

11. The method of claim 1 ,
wherein said defining comprises defining a plurality of ordered execution phases based on a plurality of policies invoked based on content of the search query,
wherein the first portion of the search process is a first subset of the plurality of ordered execution phases, and
wherein the second portion of the search process is a second subset of the plurality of ordered execution phases.

12. The method of claim 1 , wherein said defining comprises:
defining the first portion of the search process as a first plurality of ordered execution phases executable by the plurality of indexers to obtain the partial search results, and
defining the second portion of the search process as a second plurality of ordered execution phases executable by the plurality of intermediary nodes including the intermediary node to produce the combination of partial search results.

13. The method of claim 1 , wherein the intermediary node is associated with a key, and wherein the intermediary node receives partial search results exclusively associated with the key from each of the plurality of indexers.

14. The method of claim 1 , wherein the search process defines a plurality of keys for mapping a plurality of partial search results to the plurality of intermediary nodes such that each indexer sends partial search results to an intermediary node having a key that matches the key of the partial search results.

15. The method of claim 1 ,
wherein the intermediary node is a first intermediary node,
wherein the combination of partial search results is a second combination of partial search results, and
wherein the search process defines a first combination operation for the first intermediary node to produce a first combination of partial search results and defines a second combination operation for a second intermediary node to produce the second combination of partial search results based on the first combination of partial search results.

16. The method of claim 1 ,
wherein the intermediary node belongs to a plurality of first-level intermediary nodes,
wherein the combination of partial search results is a second combination of partial search results, and
wherein the search process defines a first combination operation for the plurality of first-level intermediary nodes to produce a first combination of partial search results and defines a second combination operation for a plurality of second-level intermediary nodes to produce the second combination of partial search results based on the first combination of partial search results.

17. The method of claim 1 , wherein the intermediary node is included in a hierarchy of intermediary nodes operable to produce combinations of partial search results including the combination of partial search results.

18. The method of claim 1 , further comprising:
prior to outputting the final search results,
combining, by the data intake and query system, the combination of partial search results to produce the final search results.

19. The method of claim 1 , wherein the combination of partial search results are the final search results.

20. The method of claim 1 , further comprising:
prior to receiving the combination of partial search results,
receiving, by the data intake and query system, a plurality of portions of the combination of partial search results; and
causing, by the data intake and query system, display of the plurality of portions of the combination of partial search results.

21. The method of claim 1 , further comprisi

Assignees

• Splunk Inc

Inventors

• Pal Sourav
• Mathew Ashish
• Wang Xiaowei
• Pride Christopher

Classifications

• G06F16/24532

  of parallel queries · CPC title

• G06F16/22

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F16/2471

  Distributed queries · CPC title

• G06F16/248 Primary

  Presentation of query results · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title