Method and system for implementing collection-wise processing in a log analytics system

Patent metadata

Field	Value
Publication number	US-10891297-B2
Application number	US-201615089129-A
Country	US
Kind code	B2
Filing date	Apr 1, 2016
Priority date	Apr 3, 2015
Publication date	Jan 12, 2021
Grant date	Jan 12, 2021

Abstract

Disclosed is a system, method, and computer program product for implementing a log analytics method and system that can configure, collect, and analyze log records in an efficient manner. In addition, a method and system is provided for implementing collection-wise processing within a log analytics system.

First claim

What is claimed is:

1. A method comprising:
responsive to receiving log data comprising a plurality of log entries:
storing a first copy of the log data in an indexed data store including a plurality of partitions;
indexing the log data in the indexed data store based on a plurality of temporal ranges associated, respectively, with the plurality of partitions;
grouping the plurality of log entries based on parameters associated with the log data to generate a second copy of the log data, wherein the parameters comprises two or more of:
a first set of identifiers identifying respective tenants associated with the plurality of log entries;
a second set of identifiers identifying respective targets, within a computing environment, from which the plurality of log entries are obtained; and
a third set of identifiers identifying respective sources in which the respective targets store the plurality of log entries;

storing the second copy of the log data, which is grouped based on the parameters, in a historical data store;
wherein the historical data store further comprises historical log data associated with times older than the plurality of temporal ranges in the indexed data store;
wherein the indexed data store and the historical data store are separate;

executing a first log query at least by:
determining a first temporal range associated with the first log query;
responsive to determining that the first temporal range is within one or more temporal ranges of the plurality of temporal ranges:
selecting the indexed data store rather than the historical data store for executing the first log query;
applying the first log query to one or more partitions, in the plurality of partitions, that correspond to the one or more temporal ranges;

executing a second log query at least by:
determining a second temporal range associated with the second log query;
responsive to determining that at least a first portion of the second temporal range is not within the plurality of temporal ranges:
selecting the historical data store rather than the indexed data store for executing the second log query;
applying the second log query to the historical data store;

wherein the method is performed by at least one device comprising a hardware processor.

2. The method of claim 1 , further comprising:
responsive to determining that a particular partition in the plurality of partitions has reached a size threshold: closing a particular partition, wherein the size threshold corresponds to a count of entries within the particular partition.

3. The method of claim 1 , wherein a particular partition in the plurality of partitions comprises a buffer storage portion to hold late arriving log data.

4. The method of claim 1 , wherein applying the first log query to the one or more partitions that correspond to the one or more temporal ranges comprises:
applying the query to each partition in the one or more partitions individually; and
returning query results separately for each partition.

5. One or more non-transitory machine-readable media storing instructions which, when executed by one or more processors, cause:
responsive to receiving log data comprising a plurality of log entries:
storing a first copy of the log data in an indexed data store including a plurality of partitions;
indexing the log data in the indexed data store based on a plurality of temporal ranges associated, respectively, with the plurality of partitions;
grouping the plurality of log entries based on parameters associated with the log data to generate a second copy of the log data, wherein the parameters comprises two or more of:
a first set of identifiers identifying respective tenants associated with the plurality of log entries;
a second set of identifiers identifying respective targets, within a computing environment, from which the plurality of log entries are obtained; and
a third set of identifiers identifying respective sources in which the respective targets store the plurality of log entries;

storing the second copy of the log data, which is grouped based on the parameters, in a historical data store;
wherein the historical data store further comprises historical log data associated with times older than the plurality of temporal ranges in the indexed data store;
wherein the indexed data store and the historical data store are separate;

executing a first log query at least by:
determining a first temporal range associated with the first log query;
responsive to determining that the first temporal range is within one or more temporal ranges of the plurality of temporal ranges:
selecting the indexed data store rather than the historical data store for executing the first log query;
applying the first log query to one or more partitions, in the plurality of partitions, that correspond to the one or more temporal ranges;

executing a second log query at least by:
determining a second temporal range associated with the second log query;
responsive to determining that at least a first portion of the second temporal range is not within the plurality of temporal ranges:
selecting the historical data store rather than the indexed data store for executing the second log query;
applying the second log query to the historical data store.

6. The one or more media of claim 5 , further storing instructions which, when executed by one or more processors, cause:
responsive to determining that a particular partition in the plurality of partitions has reached a size threshold: closing a particular partition, wherein the size threshold corresponds to a count of entries within the particular partition.

7. The one or more media of claim 5 , wherein a particular partition in the plurality of partitions comprises a buffer storage portion to hold late arriving log data.

8. The one or more media of claim 5 , wherein applying the first log query to the one or more partitions that correspond to the one or more temporal ranges comprises:
applying the query to each partition in the one or more partitions individually; and
returning query results separately for each partition.

9. A system, comprising:
one or more devices including a hardware processor;
the system being configured to perform operations comprising:
responsive to receiving log data comprising a plurality of log entries:
storing a first copy of the log data in an indexed data store including a plurality of partitions;
indexing the log data in the indexed data store based on a plurality of temporal ranges associated, respectively, with the plurality of partitions;
grouping the plurality of log entries based on parameters associated with the log data to generate a second copy of the log data, wherein the parameters comprises two or more of:
a first set of identifiers identifying respective tenants associated with the plurality of log entries;
a second set of identifiers identifying respective targets, within a computing environment, from which the plurality of log entries are obtained; and
a third set of identifiers identifying respective sources in which the respective targets store the plurality of log entries;

storing the second copy of the log data, which is grouped based on the parameters, in a historical data store;
wherein the historical data store further comprises historical log data associated with times older than the plurality of temporal ranges in the indexed data store;
wherein the indexed data store and the historical data store are separate;

executing a first log query at least by:
determining a first temporal range associated with the first log query;
responsive to determining that the first temporal range is within one or more temporal ra

Assignees

• Oracle Int Corp

Inventors

• Zhu Shushuai
• Akali Harish
• Russell Jerry
• He Haobo
• Royal Gaylen
• Owen Wyn

Classifications

• H04L41/5074

  Handling of user complaints or trouble tickets · CPC title

• H04L41/145

  involving simulating, designing, planning or modelling of a network · CPC title

• G06F16/1815

  Journaling file systems · CPC title

• G06F16/156

  Query results presentation · CPC title

• G06F16/148

  File search processing · CPC title