Root cause identification of a problem in a distributed computing system using log files

Patent metadata

Field	Value
Publication number	US-2021191798-A1
Application number	US-201916718707-A
Country	US
Kind code	A1
Filing date	Dec 18, 2019
Priority date	Dec 18, 2019
Publication date	Jun 24, 2021
Grant date	—

Abstract

Automated methods and systems described directed to determining a root cause of problem with a system executing in a distributed computing system. Methods and systems train a normal-state model that characterizes a normal state of the system based on normal log files generated by event sources of the system executed under normal or test conditions. Methods and systems use the normal-state model and a log file containing log messages recorded about the time when a problem with the system has been detected to identify log messages that describe a root cause of the problem.

First claim

1 . A method stored in one or more data-storage devices and executed using one or more processors of a computer system for determining a root cause of a problem with execution of a tenant's system in a distributed computing system, the method comprising:
determining a normal-state model based on relevant tokens recorded in log messages of normal log files associated with the tenant's system; determining relevant term frequencies of relevant tokens of problem-related log messages of a problem log file associated with the tenant's system; determining a message score for each problem-related log message of the problem log file based on the normal-state model and the relevant term frequencies; and rank ordering the problem-related log messages based on the message scores, wherein highest ranked problem-related log messages potentially describe the root cause of the problem with execution of the tenant's system.

2 . The method of claim 1 wherein determining the normal-state model based on the relevant tokens recorded in the log messages of the normal log files comprises:
extracting relevant tokens from log messages of each normal log file;
for each relevant token
computing a number of normal log files that contain the relevant token, and
computing an inverse document frequency value of the relevant token based on the number of normal log files that contain the relevant token; and

computing a normalized inverse document frequency value for each relevant token based on the inverse document frequency values.

3 . The method of claim 1 wherein determining the relevant term frequencies of the relevant tokens of the problem-related log messages of the problem log file comprises:
identifying problem-related log messages of the problem log file;
extracting relevant tokens from the problem-related log messages of the problem log file.
computing a frequency for each relevant token;
determining a maximum frequency of the frequencies; and
computing a relevant term frequency value for each relevant token.

4 . The method of claim 1 wherein determining the message score for each problem-related log message of the problem log file comprises:
assigning consecutive line numbers to each log message of the problem log file beginning with the log message having an oldest time stamp and ending with a most recent log message recorded in the problem log file; and
for each problem-related log message of the problem log file
aggregating relevant term frequency-inverse domain frequency values of relevant tokens, and
computing a message score based on the aggregated relevant term frequency-inverse domain frequency values.

5 . The method of claim 1 wherein determining the message score for each problem-related log message of the problem log file comprises:
assigning consecutive line numbers to each log message of the problem log file beginning with the log message having an oldest time stamp and ending with a most recent log message recorded in the problem log file;
determining a time stamp of a problem-related log message of the problem log file located closest to a suspected time when the problem occured; and
for each problem-related log message of the problem log file
aggregating relevant term frequency-inverse domain frequency values of relevant tokens,
computing a message weight based the line number of the problem-related log message and the time stamp, and
computing a message score based on the aggregated relevant term frequency-inverse domain frequency value.

6 . The method of claim 1 further comprising displaying the problem-related log messages of the problem log file in a graphical-user interface with highest ranked problem-related log messages identified as describing a potential root cause of the problem.

7 . A computer system determining a root cause of a problem with execution of a tenant's system in a distributed computing system, the system comprising:
one or more processors; one or more data-storage devices; and machine-readable instructions stored in the one or more data-storage devices that when executed using the one or more processors controls the system to perform the operations comprising:
determining a normal-state model based on relevant tokens recorded in log messages of normal log files associated with the tenant's system;
determining relevant term frequencies of relevant tokens of problem-related log messages of a problem log file associated with the tenant's system;
determining a message score for each problem-related log message of the problem log file based on the normal-state model and the relevant term frequencies; and
rank ordering the problem-related log messages based on the message scores, wherein highest ranked problem-related log messages potentially describe the root cause of the problem with execution of the tenant's system.

8 . The computer system of claim 7 wherein determining the normal-state model based on the relevant tokens recorded in the log messages of the normal log files comprises:
extracting relevant tokens from log messages of each normal log file;
for each relevant token
computing a number of normal log files that contain the relevant token, and
computing an inverse document frequency value of the relevant token based on the number of normal log files that contain the relevant token; and

computing normalized inverse document frequency values for each relevant token based on the inverse document frequency values.

9 . The computer system of claim 7 wherein determining the relevant term frequencies of the relevant tokens of the problem-related log messages of the problem log file comprises:
identifying problem-related log messages of the problem log file;
extracting relevant tokens from the problem-related log messages of the problem log file.
computing a frequency for each relevant token;
determining a maximum frequency of the frequencies; and
computing a relevant term frequency value for each relevant token.

10 . The computer system of claim 7 wherein determining the message score for each problem-related log message of the problem log file comprises:
assigning consecutive line numbers to each log message of the problem log file beginning with the log message having an oldest time stamp and ending with a most recent log message recorded in the problem log file; and
for each problem-related log message of the problem log file
aggregating relevant term frequency-inverse domain frequency values of relevant tokens, and
computing a message score based on the aggregated relevant term frequency-inverse domain frequency values.

11 . The computer system of claim 7 wherein determining the message score for each problem-related log message of the problem log file comprises:
assigning consecutive line numbers to each log message of the problem log file beginning with the log message have an oldest time stamp and ending with a most recent log message recorded in the problem log file;
determining a time stamp of a problem-related log message of the problem log file located closest to a suspected time when the problem occurred; and
for each problem-related log message of the problem log file
aggregating relevant term frequency-inverse domain frequency values of relevant tokens,
computing a message weight based the line number of the problem-related log message and the time stamp, and
computing a message score based on the aggregated relevant term frequency-inverse domain frequency value.

12 . The computer system of claim 7 further comprising displaying the problem-related log messages of the problem log file in a graphical-user interface with highest ranked problem rela

Assignees

• Vmware Inc

Inventors

• Zhang Kate
• Wang Dexiang
• Hu Michael
• Ye Tengyuan
• Miralles Eduard Serra

Classifications

• G06F11/0709 Primary

  in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems · CPC title

• G06F11/079 Primary

  Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36) · CPC title

• G06F11/0778

  Dumping, i.e. gathering error/state information after a fault for later diagnosis · CPC title

• G06F11/3476

  Data logging (G06F11/14, G06F11/2205 take precedence) · CPC title

• G06F11/0775

  Content or structure details of the error report, e.g. specific table structure, specific error fields · CPC title