Enhanced search performance using data model summaries stored in a remote data store

Patent metadata

Field	Value
Publication number	US-11892976-B2
Application number	US-202117163047-A
Country	US
Kind code	B2
Filing date	Jan 29, 2021
Priority date	Jan 29, 2021
Publication date	Feb 6, 2024
Grant date	Feb 6, 2024

Abstract

Embodiments described herein facilitate enhancement of data model acceleration, including generating data model summaries and performing searches in an accelerated manner. In one implementation, obtaining a search query from a user device. A determination may be made to execute a search, in association with the search query, via an external computing service. As such, the search query, or a variant thereof, can be provided to the external computing service, wherein the external computing service executes the search using data model summaries stored in a remote data store that is separate from a set of events from which the data model summaries were generated. A set of search results are received from the external computing service, and such search results are provided to the user device.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
obtaining, at a data-processing system, a search query from a user device;
determining, at the data-processing system, to execute a search, in association with the search query, via an external computing service;
providing the search query, or a variant thereof, to the external computing service, thereby triggering the external computing service to execute the search using data model summaries stored in a remote data store associated with the external computing service, the external computing service and the remote data store being separate from the data-processing system that generated the data model summaries and a local data store, of the data-processing system, that stores a set of events from which the data model summaries were generated;
receiving, at the data-processing system, a set of search results from the external computing service; and
providing the set of search results to the user device.

2. The computer-implemented method of claim 1 , wherein the data model summaries are stored in an optimized row columnar (ORC) file format.

3. The computer-implemented method of claim 1 , wherein the variant of the search query comprises a new search query generated to be compatible with the external computing service.

4. The computer-implemented method of claim 1 , wherein the remote data store stores different data model summaries of the data model summaries in different directories that correspond with different time and/or date based partitions of the set of events.

5. The computer-implemented method of claim 1 , wherein the external computing service is configured to execute the search of the remote data store based at least on accessing metadata from a metadata table.

6. The computer-implemented method of claim 1 wherein each of the data model summaries includes a subset of attributes, or fields, from raw data of the set of events.

7. The computer-implemented method of claim 1 , wherein the search query comprises a search command to perform statistical analysis.

8. The computer-implemented method of claim 1 , wherein determining to execute the search via the external computing service comprises analyzing an argument of the search query indicating whether to perform searches using the set of events or the data model summaries.

9. The computer-implemented method of claim 1 , wherein determining to execute the search via the external computing service comprises:
analyzing an argument of the search query indicating whether to perform searches using the set of events or the data model summaries; and
analyzing an argument of the search query indicating that the data model summaries are stored in the remote data store.

10. The computer-implemented method of claim 1 , further comprising generating the variant of the search query using a structured query language (SQL).

11. The computer-implemented method of claim 1 further comprising:
determining to provide the search query, or the variant thereof, to an indexer to execute the search using events stored in a local data store associated with the indexer; and
providing the search query to the indexer.

12. The computer-implemented method of claim 1 , wherein the remote data store resides in the external computing service on a different local area network than a search head that obtains the search query.

13. The computer-implemented method of claim 1 , wherein each event in the set of events comprises a time-stamped portion of raw machine data, the raw machine data produced by one or more components within an information technology or security environment and reflects activity within the information technology or security environment.

14. A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
obtaining, at a data-processing system, a search query from a user device;
determining, at the data-processing system, to execute a search, in association with the search query, via an external computing service;
providing the search query, or a variant thereof, to the external computing service, thereby triggering the external computing service to execute the search using data model summaries stored in a remote data store associated with the external computing service, the external computing service and the remote data store being separate from the data-processing system that generated the data model summaries and a local data store, of the data-processing system, that stores a set of events from which the data model summaries were generated;
receiving, at the data-processing system, a set of search results from the external computing service; and
providing the set of search results to the user device.

15. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:
obtaining, at a data-processing system, a search query from a user device;
determining, at the data-processing system, to execute a search, in association with the search query, via an external computing service;
providing the search query, or a variant thereof, to the external computing service, thereby triggering the external computing service to execute the search using data model summaries stored in a remote data store associated with the external computing service, the external computing service and the remote data store being separate from the data-processing system that generated the data model summaries and a local data store, of the data-processing system, that stores a set of events from which the data model summaries were generated;
receiving, at the data-processing system, a set of search results from the external computing service; and
providing the set of search results to the user device.

16. The non-transitory computer-readable medium of claim 15 , wherein the external computing service is configured to execute the search of the remote data store based at least on accessing metadata from a metadata table.

17. The non-transitory computer-readable medium of claim 15 , wherein determining to execute the search via the external computing service comprises analyzing an argument of the search query indicating whether to perform searches using the set of events or the data model summaries.

18. The non-transitory computer-readable medium of claim 15 , wherein determining to execute the search via the external computing service comprises:
analyzing an argument of the search query indicating whether to perform searches using the set of events or the data model summaries; and
analyzing an argument of the search query indicating that the data model summaries are stored in the remote data store.

19. The computer-implemented method of claim 1 , wherein a global marker file is stored at the remote data store, the global marker file including markers indicating data that has been summarized in the data model summaries.

20. The computer-implemented method of claim 1 , wherein the data model summaries are initially stored in a staging directory in the remote data store prior to being moved to a final directory in the remote data store.

Assignees

• Splunk Inc

Inventors

• Batsakis Alexandros
• Jain Ankit
• Jose Manu
• Pan Jonah
• Yan Hailun

Classifications

• G06F16/144 Primary

  Query formulation · CPC title

• G06F16/156

  Query results presentation · CPC title

• G06F16/1824

  implemented using Network-attached Storage [NAS] architecture (distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097) · CPC title

• G06F16/738

  Presentation of query results · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title