Facilitating generation of data model summaries

Patent metadata

Field	Value
Publication number	US-11841827-B2
Application number	US-202117163039-A
Country	US
Kind code	B2
Filing date	Jan 29, 2021
Priority date	Jan 29, 2021
Publication date	Dec 12, 2023
Grant date	Dec 12, 2023

Abstract

Embodiments described herein facilitate enhancement of data model acceleration, including generating data model summaries and performing searches in an accelerated manner. In one implementation, a set of events are indexed, each of the events having a corresponding index time representing a time at which the event was indexed in an indexer. Index time parameters including an index earliest time indicating a first index time at which to begin generating a data model summary and an index latest time indicating a second index time at which to complete generating the data model summary are obtained. Thereafter, a data model summary is generated. Such a data model summary summarizes events having corresponding index times between the index earliest time and the index latest time. The data model summary is provided to a remote data store that is separate from the indexer at which at least a portion of the events were indexed.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
indexing a set of events, each of the events having a corresponding index time representing a time at which the event was indexed in an indexer;
identifying, at a search head, an indication to generate a data model summary for a data model;
obtaining index time parameters including an index earliest time indicating a first index time at which to begin generating the data model summary and an index latest time indicating a second index time at which to complete generating the data model summary, the first index time and the second index time comprising index times corresponding with the events of the set of events;
generating the data model summary summarizing events having corresponding index times between the index earliest time and the index latest time; and
providing the data model summary to a remote data store for performing a subsequent search using the data model summary, wherein the remote data store that is separate from the search head and the indexer at which at least a portion of the events were indexed.

2. The computer-implemented method of claim 1 wherein the index time parameters are obtained from the search head that determines the index time parameters using event time parameters, index markers, and/or a summarization maximum interval.

3. The computer-implemented method of claim 1 , wherein the index earliest time comprises a marker latest time indicating a last index time associated with an event summarized in a previous data model summary and the index latest time comprises the marker latest time plus a summarization maximum interval indicating a maximum amount of time to use in generating the data model summary.

4. The computer-implemented method of claim 1 , wherein the index earliest time comprises an earliest event time to be included in the data model summary for the data model and the index latest time comprises the earliest event time plus a summarization maximum interval indicating a maximum amount of time to use in generating the data model summary.

5. The computer-implemented method of claim 1 further comprising:
identifying a set of buckets having events associated with the index earliest time through the index latest time; and
using the events in the set of buckets to generate the data model summary.

6. The computer-implemented method of claim 1 further comprising:
obtaining event time parameters including an event earliest time indicating a first event time for generating the data model summary and an event latest time indicating a second event time for generating the data model summary; and
using the event time parameters to generate the data model summary.

7. The computer-implemented method of claim 1 further comprising:
obtaining a staging directory path representing a staging directory at which to store the data model summary, the staging directory comprising a temporary directory used to store the data model summary during processing before data is written to a final directory; and
providing the data model summary to the staging directory at the remote data store based on the staging directory path.

8. The computer-implemented method of claim 1 further comprising:
providing the data model summary to a staging directory at the remote data store the staging directory comprising a temporary directory used during processing; and
providing a summary completion indicator to the search head, wherein the search head moves the data model summary from the staging directory to a final directory at the remote data store.

9. The computer-implemented method of claim 1 , wherein the data model summary is generated in an optimized row columnar (ORC) file format.

10. The computer-implemented method of claim 1 further comprising, at the search head:
creating a staging directory to initially host the data model summary; and
obtaining index markers from the remote data store, the index markers indicating events summarized in a previous data model summary, wherein the index markers are used to determine the index time parameters.

11. The computer-implemented method of claim 1 further comprising, at the search head:
creating a staging directory to initially host the data model summary;
obtaining index markers from the remote data store, the index markers indicating events summarized in a previous data model summary, wherein the index markers are used to determine the index time parameters; and
generating a summarization request that includes a staging directory path and the index time parameters.

12. The computer-implemented method of claim 1 further comprising, at the search head:
receiving a summary completion indicator at the search head;
initiating moving the data model summary from a staging directory to a final directory at the remote data store; and
updating a marker file with a marker earliest time and a marker latest time associated with the data model summary moved to the final directory.

13. The computer-implemented method of claim 1 , wherein the remote data store resides in an external computing service on a different local area network than the indexer.

14. The computer-implemented method of claim 1 , wherein each event in the set of events comprises a time-stamped portion of raw machine data, the raw machine data produced by one or more components within an information technology or security environment and reflects activity within the information technology or security environment.

15. A computing device, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations including:
indexing a set of events, each of the events having a corresponding index time representing a time at which the event was indexed in an indexer;
identifying, at a search head, an indication to generate a data model summary for a data model;
obtaining index time parameters including an index earliest time indicating a first index time at which to begin generating the data model summary and an index latest time indicating a second index time at which to complete generating the data model summary, the first index time and the second index time comprising index times corresponding with the events of the set of events;
generating the data model summary summarizing events having corresponding index times between the index earliest time and the index latest time; and
providing the data model summary to a remote data store for performing a subsequent search using the data model summary, wherein the remote data store that is separate from the search head and the indexer at which at least a portion of the events were indexed.

16. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to perform operations including:
indexing a set of events, each of the events having a corresponding index time representing a time at which the event was indexed in an indexer;
identifying, at a search head, an indication to generate a data model summary for a data model;
obtaining index time parameters including an index earliest time indicating a first index time at which to begin generating the data model summary and an index latest time indicating a second index time at which to complete generating the data model summary, the first index time and the second index time comprising index times corresponding with the events of the set of events;
generating the data model summary summarizin

Assignees

• Splunk Inc

Inventors

• Batsakis Alexandros
• Jain Ankit
• Jose Manu
• Pan Jonah
• Yan Hailun

Classifications

• G06F16/13 Primary

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title

• G06F16/1824

  implemented using Network-attached Storage [NAS] architecture (distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097) · CPC title

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title

• G06F16/958

  Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking · CPC title