Categorizing log records at run-time

Patent metadata

Field	Value
Publication number	US-10839308-B2
Application number	US-201514980963-A
Country	US
Kind code	B2
Filing date	Dec 28, 2015
Priority date	Dec 28, 2015
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

Methods, systems, and computer program products for categorizing log records at run-time are provided herein. A computer-implemented method includes generating one or more template signatures to be associated with each of multiple templates, wherein each of the multiple templates comprises a concatenation of one or more words; processing each of multiple log records derived from a data stream to determine a composition of each of the multiple log records; matching one or more of the generated template signatures to each of the multiple log records based on the determined composition of each of the multiple log records; and outputting an identification of (i) each of the multiple log records and (ii) the one or more generated template signatures matched thereto.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
generating, based on log data records associated with at least one given source, one or more template signatures to be associated with each of multiple templates, wherein each of the multiple templates comprises a concatenation of one or more words and one or more parameters, and wherein said generating the one or more template signatures comprises (i) identifying which particular word appears at which particular location of each of the multiple templates and (ii) ensuring that the one or more template signatures to be associated with each respective one of the multiple templates are unique;
processing each of multiple additional log records, from the same at least one given source, derived from a data stream to determine a composition of each of the multiple additional log records;
matching one or more of the generated template signatures to each of the multiple additional log records based on the determined composition of each of the multiple additional log records;
assigning unique template identifier values to the multiple additional log records, each unique template identifier value corresponding to one of the multiple templates, wherein the unique template identifier values categorize said multiple additional log records at ingestion time:
determining a frequency with which the one or more generated template signatures are matched to at least one of the multiple additional log records; and
outputting an identification of (i) each of the multiple additional log records, (ii) the one or more generated template signatures matched thereto, and (iii) the determined frequency for each of the one or more generated template signatures;
wherein the method is carried out by at least one computing device.

2. The computer-implemented method of claim 1 , wherein each of the one or more template signatures comprises a predetermined length.

3. The computer-implemented method of claim 1 , wherein said multiple templates are derived from a database.

4. The computer-implemented method of claim 1 , wherein each of the multiple templates comprises a concatenation of one or more words that pertain to system and/or application records.

5. The computer-implemented method of claim 1 , wherein the data stream is obtained via a data center.

6. The computer-implemented method of claim 1 , wherein the composition of each of the multiple additional log records comprises an arrangement of one or more words.

7. The computer-implemented method of claim 1 , wherein said outputting comprises outputting the identification to a database.

8. The computer-implemented method of claim 1 , wherein said outputting comprises outputting the identification to a user.

9. The computer-implemented method of claim 1 , wherein said determining the frequency comprises learning a workload distribution associated with the data stream.

10. The computer-implemented method of claim 1 , wherein said determining the frequency further comprises maintaining a cache comprising each of one or more of the generated template signatures having a determined frequency above a predetermined threshold.

11. A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:
generate, based on log data records associated with at least one given source, one or more template signatures to be associated with each of multiple templates, wherein each of the multiple templates comprises a concatenation of one or more words and one or more parameters, and wherein said generating the one or more template signatures comprises (i) identifying which particular word appears at which particular location of each of the multiple templates and (ii) ensuring that the one or more template signatures to be associated with each respective one of the multiple templates are unique;
process each of multiple additional log records, from the same at least one given source, derived from a data stream to determine a composition of each of the multiple additional log records;
match one or more of the generated template signatures to each of the multiple additional log records based on the determined composition of each of the additional multiple log records;
assign unique template identifier values to the multiple additional log records, each unique template identifier value corresponding to one of the multiple templates, wherein the unique template identifier values categorize said multiple additional log records at ingestion time:
determine a frequency with which the one or more generated template signatures are matched to at least one of the multiple additional log records; and
output an identification of (i) each of the multiple additional log records, (ii) the one or more generated template signatures matched thereto, and (iii) the determined frequency for each of the one or more generated template signatures.

12. The computer program product of claim 11 , wherein each of the one or more template signatures comprises a predetermined length.

13. The computer program product of claim 11 , wherein each of the multiple templates comprises a concatenation of one or more words that pertain to system and/or application records.

14. The computer program product of claim 11 , wherein said outputting comprises outputting to a database and/or a user.

15. The computer program product of claim 11 , wherein said determining the frequency comprises learning a workload distribution associated with the data stream.

16. The computer program product of claim 11 , wherein said determining the frequency further comprises maintaining a cache comprising each of one or more of the generated template signatures having a determined frequency above a predetermined threshold.

17. A system comprising:
a memory; and
at least one processor coupled to the memory and configured for:
generating, based on log data records associated with at least one given source, one or more template signatures to be associated with each of multiple templates, wherein each of the multiple templates comprises a concatenation of one or more words and one or more parameters, and wherein said generating the one or more template signatures comprises (i) identifying which particular word appears at which particular location of each of the multiple templates and (ii) ensuring that the one or more template signatures to be associated with each respective one of the multiple templates are unique;
processing each of multiple additional log records, from the same at least one given source, derived from a data stream to determine a composition of each of the multiple additional log records;
matching one or more of the generated template signatures to each of the multiple additional log records based on the determined composition of each of the multiple additional log records;
assigning unique template identifier values to the multiple additional log records, each unique template identifier value corresponding to one of the multiple templates, wherein the unique template identifier values categorize said multiple additional log records at ingestion time:
determining a frequency with which the one or more generated template signatures are matched to at least one of the multiple additional log records; and
outputting an identification of (i) each of the multiple additional log records, (ii) the one or more generated template signatures matched thereto, and (iii) the determined frequency for each of the one or more

Assignees

• IBM

Inventors

• Agarwal Manoj K
• Deshpande Prasad M

Classifications

• G06F16/3344 Primary

  using natural language analysis · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06F16/24

  Querying · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title

• G06F16/355

  Creation or modification of classes or clusters · CPC title