Indexing and searching log records using templates index and attributes index

Patent metadata

Field	Value
Publication number	US-2016034510-A1
Application number	US-201414445232-A
Country	US
Kind code	A1
Filing date	Jul 29, 2014
Priority date	Jul 29, 2014
Publication date	Feb 4, 2016
Grant date	—

Abstract

A computer partitions a log stream into records having defined intervals. For each of the records, the computer selects a template from among a set of templates in a templates index repository based on an identifier for the record being associated in the templates index repository with the template, identifies an attribute contained in the record based on the template, and stores the attribute associated with an identifier for the record and an identifier for the template in an attributes index repository. The attributes index repository and the templates index repository are used to retrieve records responsive to search queries.

First claim

1 . A method by a computer comprising:
partitioning a log stream into records having defined intervals; and for each of the records, performing:
selecting a template from among a set of templates in a templates index repository based on an identifier for the record being associated in the templates index repository with the template;
identifying an attribute contained in the record based on the template; and
storing the attribute associated with an identifier for the record and an identifier for the template in an attributes index repository.

2 . The method of claim 1 , further comprising:
repeating for each of a plurality of log streams from a plurality of software sources executed by host nodes, the partitioning, the selecting, the identifying, and the storing.

3 . The method of claim 1 , further comprising:
receiving a search query containing a search term; identifying a plurality of the templates from among the set of templates in the templates index repository based on content of the search query; and for each of the plurality of the templates that were identified, performing:
identifying one of the attributes contained in the attributes index repository that is associated with an identifier for the template and satisfies the search term;
obtaining from the attributes index repository the identifier for the record associated with the one of the attributes;
retrieving the record using the identifier for the record that was obtained from the attributes index repository; and
outputting the record as a response to the search query.

4 . The method of claim 3 , wherein identifying a plurality of the templates from among the set of templates in the templates index repository based on content of the search query, comprises:
identifying templates in the templates index repository associated with defined intervals that at least partially overlap an interval identified by the search query.

5 . The method of claim 4 , wherein:
the log stream is partitioned into records having a defined time period; receiving the search query comprises identifying a search time period and identifying a log stream source; and the plurality of the templates are identified from among the set of templates in the templates index repository based on the search time period and the log stream source.

6 . The method of claim 3 , further comprising:
identifying a plurality of search terms contained in the search query; ranking the records identified for the plurality of the templates based on how many of the search terms are satisfied by the respective records; and filtering which of the records are output based on the ranking.

7 . The method of claim 1 , wherein identifying the attribute contained in the record based on the template, comprises:
determining a location of the attribute within the record based on the template.

8 . The method of claim 1 , further comprising:
determining that a template does not exist in the templates index repository for one of the records; and inserting a new template in the templates index repository associated with an identifier for the one of the records.

9 . The method of claim 8 , wherein the new template is generated based on structure of a log stream that is output by software code of a source of the log stream.

10 . The method of claim 1 , further comprising:
determining that one of the records corresponds to a new time period; and inserting a new template in the templates index repository associated with an identifier for the one of the records and an identifier for the new time period.

11 . The method of claim 1 , wherein the attribute comprises text contained in the record.

12 . The method of claim 1 , wherein:
software that performs queries of the templates index repository is separate from software that performs queries of the attributes index repository.

13 . The method of claim 1 , wherein:
the search query is received from a user equipment; and the record is output toward the user equipment.

14 . A computer program product comprising:
a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code to partition a log stream into records having defined intervals;
computer readable program code to, for each of the records, perform:
selecting a template from among a set of templates in a templates index repository based on an identifier for the record being associated in the templates index repository with the template;
identifying an attribute contained in the record based on the template; and
storing the attribute associated with an identifier for the record and an identifier for the template in an attributes index repository.

15 . The computer program product of claim 14 , the computer readable program code further comprising:
computer readable program code to repeat for each of a plurality of log streams from a plurality of software sources executed by host nodes, the partitioning, the selecting, the identifying, and the storing;

16 . The computer program product of claim 14 , the computer readable program code further comprising:
computer readable program code to receive a search query containing a search term; computer readable program code to identify a plurality of the templates from among the set of templates in the templates index repository based on content of the search query; computer readable program code to, for each of the plurality of the templates that were identified, perform:
identifying one of the attributes contained in the attributes index repository that is associated with an identifier for the template and satisfies the search term;
obtaining from the attributes index repository the identifier for the record associated with the one of the attributes;
retrieving the record using the identifier for the record that was obtained from the attributes index repository; and
outputting the record as a response to the search query.

17 . The computer program product of claim 16 , wherein identifying a plurality of the templates from among the set of templates in the templates index repository based on content of the search query, comprises:
identifying templates in the templates index repository associated with defined intervals that at least partially overlap an interval identified by the search query.

18 . The computer program product of claim 17 , wherein:
the log stream is partitioned into records having a defined time period; receiving the search query comprises identifying a search time period and identifying a log stream source; and the plurality of the templates are identified from among the set of templates in the templates index repository based on the search time period and the log stream source.

19 . The computer program product of claim 16 , the computer readable program code further comprising:
computer readable program code to identify a plurality of search terms contained in the search query; computer readable program code to rank the records identified for the plurality of the templates based on how many of the search terms are satisfied by the respective records; and computer readable program code to filter which of the records are output based on the ranking.

20 . The computer program product of claim 14 , the computer readable program code further comprising:
computer readable program code to determine a location of the attribute within the record based o

Assignees

• Ca Inc

Inventors

• Gukal Sreenivas

Classifications

• G06F16/31 Primary

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F17/30336 Primary

  Physics · mapped topic

• G06F17/3053

  Physics · mapped topic

• G06F17/30867

  Physics · mapped topic