Method and system for identifying event-message transactions

Patent metadata

Field	Value
Publication number	US-10810103-B2
Application number	US-201615379005-A
Country	US
Kind code	B2
Filing date	Dec 14, 2016
Priority date	Dec 14, 2016
Publication date	Oct 20, 2020
Grant date	Oct 20, 2020

Abstract

The current document is directed to methods and systems that process, classify, efficiently store, and display large volumes of event messages generated in modern computing systems. In a disclosed implementation, event messages are assigned types and transformed into event records with well-defined fields that contain field values. Recurring patterns of event messages, referred to as “transactions,” are identified within streams or sequences of time-associated event messages and streams or sequences of time-associated event records.

First claim

The invention claimed is:

1. A subsystem that identifies groups of related event-message types, the subsystem comprising:
one or more processors;
one or more memories;
one or more data-storage devices; and
computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the subsystem to
receive event messages from event-message sources within a distributed computer system that includes the subsystem;
assign event-message types to the received event messages;
count event-message-type co-occurrences, in time, within the received messages;
determine and store in memory, from the counted event-message-type co-occurrences, pairs of related event-message types; and
use the determined pairs of related event-message types to select, from one or more sets of event messages, related event-message types that are output to one or more of an automated analysis subsystem, a display device, and a system monitor.

2. The subsystem of claim 1 wherein the subsystem assigns an event-message type to each received event message by:
identifying non-parameter tokens within the event messages;
determining a feature vector generated from the non-parameter tokens identified within the event message,
determining a distance between the determined feature vector and a feature vector associated with each event-message cluster, and
assigning to the event message a type associated with a group of event messages associated with the feature vector at the smallest determined distance from the determined feature vector.

3. The subsystem of claim 1 wherein the subsystem counts event-message-type co-occurrences, in time, within the received messages by:
maintaining a set of counters, each associated with two event-message types and a most-recent access time;
maintaining each received event message in a memory buffer, along with a timestamp, in time order, along with earlier and later received messages until at least one more recently received event message in the memory buffer is outside an event-message window that includes the received event message; and
counting each co-occurring event message in the memory buffer within the window that includes the received event message in a counter associated with the type of the co-occurring event message and the type associated with the received event message.

4. The subsystem of claim 3 wherein the window that includes the received event message includes additional event messages associated times that differ from the time associated with the received event message by less than a time-difference value.

5. The subsystem of claim 4 wherein the window that includes the received event message includes no more than a maximum number of additional event message.

6. The subsystem of claim 3 wherein counting each co-occurring event message in the memory buffer within the window that includes the received event message in a counter associated with the type of the co-occurring event message and the type associated with the received event message further comprises:
when one of the counters in the set of counters is associated with the type of the co-occurring event message and the type associated with the received event message,
setting the most-recent access time associated with the counter to the time represented by one of the timestamps associated with the co-occurring event message and the received event message, and
incrementing the counter; and

when no counters in the set of counters is associated with the type of the co-occurring event message and the type associated with the received event message,
reinitializing a counter in the set of counters having a most-recent access time earliest in time by
setting the most-recent access time associated with the counter to the time represented by one of the timestamps associated with the co-occurring event message and the received event message,
setting the counter to one, and
associating the counter with the type of the co-occurring event message and the type associated with the received event message.

7. The subsystem of claim 3 wherein the subsystem determines and stores in memory, from the counted event-message-type co-occurrences, pairs of related event-message types by:
for each event-message type associated with a counter,
determining a conditional probability for the co-occurrence of other event-message types associated with counters also associated with the event-message type;
generating a set of event-message-type pairs that each includes the event-message type and a different other co-occurring event-message type, the set ordered by conditional probability for the pairs; and
adding at most a fixed number of the event-message-type pairs with greatest conditional probabilities to determined and stored related event-message types.

8. The subsystem of claim 3 wherein event messages are one of:
raw event messages; and
event records that include the event-message type and the values of parameter fields.

9. A subsystem that identifies event-message transactions, the subsystem comprising:
one or more processors;
one or more memories;
one or more data-storage devices; and
computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the subsystem to
receive a set of typed and time-stamped event messages from event-message sources within a distributed computer system that includes the subsystem;
identify a set of identifier fields among parameter fields of the received set of typed and time-stamped event messages;
determine and store in memory a set of multi-event-message transactions within the set of typed and time-stamped event messages; and
use the determined multi-event-message transactions to select, from one or more sets of event messages, multi-event-message transactions that are output to one or more of an automated analysis subsystem, a display device, and a system monitor.

10. The subsystem of claim 9 wherein the subsystem assigns an event-message type to each received event message by:
identifying non-parameter tokens within the event messages;
determining a feature vector generated from the non-parameter tokens identified within the event message,
determining a distance between the determined feature vector and a feature vector associated with each event-message cluster, and
assigning to the event message a type associated with a group of event messages associated with the feature vector at the smallest determined distance from the determined feature vector.

11. The subsystem of claim 9 wherein event messages are one of:
raw event messages, values of parameter fields within which are extracted by regular expressions; and
event records that include the event-message type and the values of parameter fields that are extracted by field types and positions associated with each type of event record.

12. The subsystem of claim 9 wherein event-message transactions comprise multiple event messages the co-occur within a time window and that have common values for each of one or more sets of identifier fields.

13. The subsystem of claim 12 wherein the subsystem identifies a set of identifier fields among parameter fields of the received set of typed and time-stamped event messages by:
for each of a number of common event-message types t,
for each of a number of time windows w,
for each of a number of initial candidate identifier fields f,
extracting two sets of values from typed and time-stamped event messages of event-message type t within time window w for field f,

Assignees

• Vmware Inc

Inventors

• Lin Junyuan
• Kushmerick Nicholas
• Herlocker Jon

Classifications

• G06F11/30 Primary

  Monitoring · CPC title

• G06F11/3006

  where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50) · CPC title

• G06F2201/86

  Event-based monitoring · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title