Data fabric service system deployment

Patent metadata

Field	Value
Publication number	US-11797618-B2
Application number	US-202217810165-A
Country	US
Kind code	B2
Filing date	Jun 30, 2022
Priority date	Sep 26, 2016
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

Disclosed is a data fabric service system that can be implemented in a distributed computer network, such as a data intake and query system. The data index and query system can receive a search query and define a search scheme for applying the search query on distributed data storage systems including internal data storage and external data storage. The data index and query system may provide a portion of the search scheme to a search service of the data fabric service system, which can cause worker nodes of the data fabric service system to perform various functions—including applying the search query to the external data storage based on the portion of the search scheme in order to obtain search results.

First claim

What is claimed is:

1. A method, comprising:
defining, by a data intake and query system, a search scheme based at least in part on a query, wherein the data intake and query system comprises one or more worker nodes, a search service provider, and one or more indexers;
triggering, by the data intake and query system, the one or more indexers to search one or more internal data stores based at least in part on a first portion of the search scheme to obtain first partial search results; and
providing, by the data intake and query system, a second portion of the search scheme to the search service provider, wherein the search service provider:
defines, based at least in part on the second portion of the search scheme, a search process executable by the one or more worker nodes, and
directs the one or more worker nodes to communicate with one or more external data sources to obtain second partial search results based at least in part on the second portion of the search scheme.

2. The method of claim 1 , wherein the one or more worker nodes are configured to:
receive the search process from the search service provider, and
based on the search process, communicate with the one or more external data sources to obtain the second partial search results.

3. The method of claim 1 , wherein the one or more worker nodes are configured to:
receive the search process from the search service provider, and
based on the search process:
communicate with the one or more external data sources to obtain the second partial search results,
aggregate the second partial search results, and
provide aggregated partial search results to the search service provider.

4. The method of claim 1 , wherein the one or more worker nodes are communicatively coupled to the search service provider.

5. The method of claim 1 , wherein the data intake and query system further comprises a search head.

6. The method of claim 1 , wherein the data intake and query system further comprises a search head, wherein defining the search scheme comprises defining the search scheme by the search head.

7. The method of claim 1 , wherein the query is received from a user computing device.

8. The method of claim 1 , wherein one or more external data sources comprise one or more computing devices other than the one or more worker nodes, wherein the one or more worker nodes receive the second partial search results from the one or more computing devices using a parallel export technique.

9. The method of claim 1 , wherein the search process is a logical directed acyclic graph (DAG).

10. The method of claim 1 , wherein the one or more worker nodes are configured to process the second partial search results into time-ordered event chunks and stream the time-ordered event chunks to the search service provider.

11. The method of claim 2 , wherein the one or more external data sources store data in a structured format that is different from an events format of the one or more internal data stores.

12. The method of claim 1 , wherein the one or more internal data stores store data as a plurality of time-indexed events, each event of the plurality of time-indexed events including a portion of raw machine data associated with a timestamp.

13. The method of claim 1 , wherein the one or more indexers send the first partial search results to the one or more worker nodes.

14. The method of claim 1 , wherein the one or more indexers send the first partial search results to the one or more worker nodes, wherein the one or more worker nodes are configured to aggregate the first partial search results and the second partial search results.

15. The method of claim 1 , further comprising rendering an output of data indicative of at least one of the first partial search results or the second partial search results for display on a user interface of a display device associated with a user.

16. The method of claim 1 , wherein the one or more worker nodes are co-located with the one or more indexers.

17. The method of claim 1 , wherein the one or more indexers launch the one or more worker nodes.

18. The method of claim 1 , wherein the query is expressed in a pipelined search language.

19. A data intake and query system comprising:
a memory; and
a processing device coupled with the memory to:
define a search scheme based at least in part on a query, wherein the data intake and query system comprises one or more worker nodes, a search service provider, and one or more indexers;
trigger the one or more indexers to search one or more internal data stores based at least in part on a first portion of the search scheme to obtain first partial search results; and
provide a second portion of the search scheme to the search service provider, wherein the search service provider:
defines, based at least in part on the second portion of the search scheme, a search process executable by the one or more worker nodes, and
directs the one or more worker nodes to communicate with one or more external data sources to obtain second partial search results based at least in part on the second portion of the search scheme.

20. A non-transitory computer-readable medium encoding instructions thereon that, in response to execution by one or more processing devices of a data intake and query system, cause the one or more processing devices to:
define a search scheme based at least in part on a query, wherein the data intake and query system comprises one or more worker nodes, a search service provider, and one or more indexers;
trigger the one or more indexers to search one or more internal data stores based at least in part on a first portion of the search scheme to obtain first partial search results; and
provide a second portion of the search scheme to the search service provider, wherein the search service provider:
defines, based at least in part on the second portion of the search scheme, a search process executable by the one or more worker nodes, and
directs the one or more worker nodes to communicate with one or more external data sources to obtain second partial search results based at least in part on the second portion of the search scheme.

Assignees

• Splunk Inc

Inventors

• Pal Sourav
• Pride Christopher
• Bhattacharjee Arindam
• Wang Xiaowei
• Hodge James Alasdair Robert
• Ahamed Mustafa

Classifications

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title

• G06F16/211

  Schema design and management · CPC title

• G06F16/212

  with details for data modelling support · CPC title

• G06F16/248

  Presentation of query results · CPC title

• G06F16/2455

  Query execution · CPC title