Data intake and query system to analyze and index machine raw data from heterogenous data sources

Patent metadata

Field	Value
Publication number	US-12430332-B2
Application number	US-202418429156-A
Country	US
Kind code	B2
Filing date	Jan 31, 2024
Priority date	Jan 31, 2024
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

A data intake and query system can process a query to identify subquery tokens corresponding to subqueries to be executed by external data systems. The data intake and query system can process the subquery tokens to generate modified subqueries to be executed by the external data systems. The modified subqueries can cause the external data system to return metadata associated with the events processed by the external data systems during executing of the modified subqueries.

First claim

What is claimed:

1. A method, comprising:
receiving, at a first query system, a search query;
identifying a plurality of subquery tokens of the search query based on parsing the search query, wherein the plurality of subquery tokens are associated with a plurality of external data systems;
processing a first subquery token of the plurality of subquery tokens to generate first instructions for a first external data system, associated with the first subquery token, of the plurality of external data systems to execute a first subquery corresponding to the first subquery token;
based on the processing the first subquery token, modifying the first instructions for the first external data system to collect metadata of first events processed by the first external data system during execution of the first subquery;
communicating the modified first instructions to the first external data system;
receiving the metadata of the first events, wherein the metadata of the first events includes a first external offset for each of the first events;
mapping the first external offsets to first search query offsets of a plurality of search query offsets of the first query system, wherein the plurality of search query offsets correspond to a plurality of events used by the plurality of external data systems during execution of respective subqueries; and
causing a display to display results of the search query based at least in part on the first search query offsets.

2. The method of claim 1 , wherein modifying the first instructions for the first external data system comprises modifying the first subquery to include a search command to collect metadata associated with the first events.

3. The method of claim 1 , wherein processing the first subquery comprises determining that the first subquery includes a reporting command.

4. The method of claim 3 , wherein the reporting command indicates at least one of: a result of the first subquery is a count or a sum, and a result of the subquery does not include the first events.

5. The method of claim 3 , wherein modifying the first instructions for the first external data system comprises adding an instruction for a search head of the first external data system to collect the metadata of the first events from one or more search nodes of the first external data system during execution of the first subquery.

6. The method of claim 5 , wherein the one or more search nodes store the first events.

7. The method of claim 1 , wherein processing the first subquery comprises determining that a search head of the first external data system and at least one search node of the first external data system are co-located on a same computing device.

8. The method of claim 7 , wherein modifying the first instructions for the first external data system comprises adding an instruction for the search head to collect the metadata of the first events from the first external data system during execution of the first subquery.

9. The method of claim 1 , wherein processing the first subquery comprises determining that at least one command of the first subquery is stored by the first query system.

10. The method of claim 9 , wherein modifying the first instructions for the first external data system comprises adding an instruction for a search head of the first external data system to collect the metadata of the first events from one or more search nodes of the first external data system during execution of the first subquery.

11. The method of claim 1 , wherein processing the first subquery token of the plurality of subquery tokens comprises performing a lookup using the first subquery token to identify the first subquery and the first external data system.

12. The method of claim 1 , further comprising identifying the first external data system using the first subquery token.

13. The method of claim 1 , wherein mapping the first external offsets to the first search query offsets comprises mapping each first external offset of the first external offsets to a respective first search query offset of the first search query offsets.

14. The method of claim 1 , further comprising:
processing a second subquery token of the plurality of subquery tokens to generate second instructions for a second external data system of the plurality of external data systems to execute a second subquery, the second subquery corresponding to the second subquery token;
on the processing the second subquery token, modifying the second instructions for the second external data system to collect metadata of second events processed by the second external data system during execution of the second subquery;
communicating the modified second instructions to the second external data system;
receiving the metadata of the second events, wherein the metadata of the second events includes a second external offset for each of the second events;
mapping the second external offsets of the second events to second search query offsets of the plurality of search query offsets of the first query system; and
causing the display to display the results of the search query based at least in part on the second search query offsets.

15. The method of claim 1 , further comprising:
requesting the first events from the first query system based at least in part on the first external offsets;
receiving the first events from the first query system; and
causing the display to display at least one of the first events.

16. A system, comprising:
one or more processors of a first query system, the one or more processors configured to:
receive a search query;
identify a plurality of subquery tokens of the search query based on parsing the search query, wherein the plurality of subquery tokens are associated with a plurality of external data systems;
process a first subquery token of the plurality of subquery tokens to generate first instructions for a first external data system, associated with the first subquery token, of the plurality of external data systems to execute a first subquery corresponding to the first subquery token;
based on processing the first subquery token, modify the first instructions for the first external data system to collect metadata of first events processed by the first external data system during execution of the first subquery;
communicate the modified first instructions to the first external data system;
receive the metadata of the first events, wherein the metadata of the first events includes a first external offset for each of the first events;
map the first external offsets to first search query offsets of a plurality of search query offsets of the first query system, wherein the plurality of search query offsets correspond to a plurality of events used by the plurality of external data systems during execution of respective subqueries; and
cause a display to display results of the search query based at least in part on the first search query offsets.

17. The system of claim 16 , wherein to modify the first instructions for the first external data system, the one or more processors are configured to modify the first subquery to include a search command to collect metadata associated with the first events.

18. The system of claim 16 , wherein to map the first external offsets to the first search query offsets, the one or more processors are configured to map each first external offset of the first external offsets to a respective first search query offset of the first search query offsets.

19. A non-transitory computer readable media, compri

Assignees

• Splunk Inc

Inventors

• Frenkel Nir
• Rodrigues Galdino Marcus Vinicius
• Manivel Vinay
• Rao Balaji

Classifications

• G06F16/248

  Presentation of query results · CPC title

• G06F16/256 Primary

  in federated or virtual databases · CPC title

• G06F16/24542

  Plan optimisation · CPC title

• G06F16/24535 Primary

  of sub-queries or views · CPC title