Management of log data

Patent metadata

Field	Value
Publication number	US-2016299934-A1
Application number	US-201514683005-A
Country	US
Kind code	A1
Filing date	Apr 9, 2015
Priority date	Apr 9, 2015
Publication date	Oct 13, 2016
Grant date	—

Abstract

A system according to certain aspects improves the process of managing log data. The system may receive a log data file including one or more log lines that contain information relating to computing operations. At least some of the log lines may include a static portion and a variable portion. The system may process a first log line to identify and extract a static portion from the first log line, and determine a first value for the first log line based on the extracted static portion. The system may process a second log line to identify and extract a static portion from the second log line, and determine a second value for the second log line based on the extracted static portion. The system may compare the first and second values, and based on the comparison, organize the first and second log lines together for presentation to a user.

First claim

1 . A method for processing log data, the method comprising:
with one or more computing devices comprising computer hardware,
receiving a log data file comprising one or more log lines, the log lines including information relating to one or more computing operations, wherein at least some of the log lines include a static portion and a variable portion;
processing a first log line of the one or more log lines to identify a static portion of the first log line;
extracting the static portion from the first log line;
determining a first value for the first log line based on the extracted static portion;
processing a second log line of the one or more log lines to identify a static portion of the second log line;
extracting the static portion from the second log line;
determining a second value for the second log line based on the extracted static portion;
comparing the first and second values; and
based on the comparison, organizing the first and second log lines together for presentation to a user.

2 . The method of claim 1 , wherein the first and second values are the same and uniquely identify the static portion of text of the first log line and the second log line.

3 . The method of claim 1 , wherein said organizing comprises adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with a unique set of values that are not associated with any other group.

4 . The method of claim 1 , further comprising:
extracting a variable portion from the first log line of the one or more log lines; determining a third value for the first log line based on the extracted variable portion; extracting a variable portion from the second log line of the one or more log lines; and determining a fourth value for the second log line based on the extracted variable portion.

5 . The method of claim 4 , wherein said organizing comprises:
adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with values that are identical to the first and second values; and adding the first and second log lines to a sub-group within the group of log lines based on the third and fourth values, wherein the log lines in the sub-group are associated with values that are identical to the third and fourth values.

6 . The method of claim 1 , wherein said determining of the first and second values comprises determining the first and second values using a hash function configured to generate a value based on the extracted static portions of the first and second log lines, respectively.

7 . The method of claim 1 , wherein said determining of the first and second values comprises:
extracting a first set of one or more trigrams from the extracted static portion of the first log line and a second set of one or more trigrams from the extracted static portion of the second log line; calculating a value for each of the first and second sets of one or more trigrams; and determining the first and second values by aggregating the values calculated for the first and second sets of one or more trigrams, respectively.

8 . The method of claim 7 , wherein the values calculated for the first and second sets of one or more trigrams are aggregated by calculating a sum of the values.

9 . The method of claim 1 , wherein the static portions of the first and second log lines indicate the types of transaction associated with the first and second log lines, respectively, and variable portions of the first and second log lines indicate additional information regarding the transactions associated with the first and second log lines, respectively.

10 . The method of claim 1 , wherein said extracting of the static portion comprises determining a sequence of characters that have been processed more than a threshold number of times in the history of processing the log data.

11 . A system for processing log data, the system comprising:
a computing device comprising computer hardware and configured to: receive a log data file comprising one or more log lines, the log lines including information relating to one or more computing operations, wherein at least some of the log lines include a static portion and a variable portion; process a first log line of the one or more log lines to identify a static portion of the first log line; extract the static portion from the first log line; determine a first value for the first log line based on the extracted static portion; process a second log line of the one or more log lines to identify a static portion of the second log line; extract the static portion from the second log line; determine a second value for the second log line based on the extracted static portion; compare the first and second values; and based on the comparison, organize the first and second log lines together for presentation to a user.

12 . The system of claim 11 , wherein the first and second values are the same and uniquely identify the static portion of text of the first log line and the second log line.

13 . The system of claim 11 , wherein the computing device is configured to organize the first and second log lines at least by adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with a unique set of values that are not associated with any other group.

14 . The system of claim 11 , wherein the computing device is further configured to:
extract a variable portion from the first log line of the one or more log lines; determine a third value for the first log line based on the extracted variable portion; extract a variable portion from the second log line of the one or more log lines; and determine a fourth value for the second log line based on the extracted variable portion.

15 . The system of claim 14 , wherein the computing device is configured to organize the first and second log lines at least by:
adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with values that are identical to the first and second values; and adding the first and second log lines to a sub-group within the group of log lines based on the third and fourth values, wherein the log lines in the sub-group are associated with values that are identical to the third and fourth values.

16 . The system of claim 11 , wherein the computing device is configured to determine the first and second values at least by determining the first and second values using a hash function configured to generate a value based on the extracted static portions of the first and second log lines, respectively.

17 . The system of claim 11 , wherein the computing device is configured to determine the first and second values at least by:
extracting a first set of one or more trigrams from the extracted static portion of the first log line and a second set of one or more trigrams from the extracted static portion of the second log line; calculating a value for each of the first and second sets of one or more trigrams; and determining the first and second values by aggregating the values calculated for the first and second sets of one or more trigrams, respectively.

18 . The system of claim 17 , wherein the values calculated for the first and second sets of one or more trigrams are a

Assignees

• Commvault Systems Inc

Inventors

• Karandikar Amey Vijaykumar
• Vibhor Anand
• Upadhyay Mrityunjay

Classifications

• G06F17/3033

  Physics · mapped topic

• G06F17/30575

  Physics · mapped topic

• G06F17/30227

  Physics · mapped topic

• G06F17/30368 Primary

  Physics · mapped topic

• G06F17/30088

  Physics · mapped topic