Querying distributed log data using virtual fields defined in query strings

Patent metadata

Field	Value
Publication number	US-10713271-B2
Application number	US-201815888890-A
Country	US
Kind code	B2
Filing date	Feb 5, 2018
Priority date	Feb 10, 2017
Publication date	Jul 14, 2020
Grant date	Jul 14, 2020

Abstract

The present disclosure relates to querying log messages stored at log endpoints (e.g., log cores) using query strings that reference virtual fields, and generating log field views presenting aggregate information for groups of log messages that are grouped based on criteria defined in the query string.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving a plurality of log messages, wherein each log message of the plurality of log messages includes message data, and wherein the message data includes one or more detectable signatures to segment the message data into data fields;
mapping each log message of the plurality of log messages to a message structure, wherein the message structure defines a fixed portion or a variable portion of the message data as a data field;
detecting, for each log message of the plurality of log messages, one or more data fields within the message data, based at least in part on the one or more detectable signatures included in the message data and the mapping of the log message to the message structure, wherein each data field of the one or more data fields corresponds to a value present in the message data;
receiving, via a user interface, input corresponding to a query string for querying at least a portion of the plurality of log messages, the query string including one or more virtual data fields, wherein each of the one or more virtual data fields corresponds to a function of at least one data field of the detected one or more data fields, and wherein the one or more virtual data fields are not present in the detected one or more data fields or the message structure;
parsing the query string to identify each of the one or more virtual data fields;
in response to the parsing, formulating a response to the query string at least in part by:
querying one or more log data stores using the at least one data field, wherein the plurality of log messages are stored at the one or more log data stores,
receiving, for each of the one or more log data stores, partial result data in response to the querying of the log data store, the partial result data including one or more results of querying log messages stored at the log data store,
aggregating the partial result data received from each of the one or more log data stores to determine a preliminary result for the at least one data field, and
performing the function on the preliminary result to determine a formulated response to the query string that included the one or more virtual data fields;
generating a log field view representing the formulated response to the query string, wherein the formulated response to the query string presents the preliminary result; and
presenting, at the user interface, the log field view.

2. The computer-implemented method of claim 1 , wherein a virtual data field of the one or more virtual data fields is based on a start of a transaction or an end of the transaction, wherein the transaction corresponds to a transaction identifier, wherein the transaction identifier is a detectable field in a log message of the plurality of log messages, and wherein the start and/or the end of the transaction is computed based on a set of log messages associated with the transaction.

3. The computer-implemented method of claim 1 , wherein presenting the log field view at the interface includes presenting a table including each of the one or more virtual data fields, such that each of the one or more virtual data fields includes an associated one or more virtual data elements, and each of the one or more virtual data elements includes a value.

4. The computer-implemented method of claim 3 , further comprising:
receiving, at the interface, additional input corresponding to a selection of a virtual data element from amongst the one or more virtual data fields presented in the table; and
identifying at least one log message associated with the selected virtual data element; and
presenting information associated with each of the at least one identified log message.

5. The computer-implemented method of claim 1 , further comprising:
performing a group-by operation on the aggregated partial result data using the one or more virtual fields, wherein performing the group-by operation includes grouping log messages of the plurality of log messages into one or more groups, wherein log messages included in a group share a same attribute with other log messages in the group.

6. The computer-implemented method of claim 1 , wherein the plurality of log messages includes each of a first log message and a second log message, wherein the first log message is in a first format and the second log message is in a second format, and wherein the first format is different from the second format.

7. The computer-implemented method of claim 6 , wherein the first log message includes a particular data field at a first position within the first log message, wherein the second log message includes the particular data field at a second position within the second log message, and wherein the first position is different from the second position.

8. The computer-implemented method of claim 6 , wherein the first log message includes a particular data field represented using a first field name within the first log message, wherein the second log message includes the particular data field represented using a second field name within the second log message, and wherein the first field name is different from the second field name.

9. A system, comprising:
one or more data processors; and
a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including:
receiving a plurality of log messages, wherein each log message of the plurality of log messages includes message data, and wherein the message data includes one or more detectable signatures to segment the message data into data fields;
mapping each log message of the plurality of log messages to a message structure, wherein the message structure defines a fixed portion or a variable portion of the message data as a data field;
detecting, for each log message of the plurality of log messages, one or more data fields within the message data, based at least in part on the one or more detectable signatures included in the message data and the mapping of the log message to the message structure, wherein each data field of the one or more data fields corresponds to a value present in the message data;
receiving, via a user interface, input corresponding to a query string for querying at least a portion of the plurality of log messages, the query string including one or more virtual data fields, wherein each of the one or more virtual data fields corresponds to a function of at least one data field of the detected one or more data fields, and wherein the one or more virtual data fields are not present in the detected one or more data fields or the message structure,
parsing the query string to identify each of the one or more virtual data fields;
in response to the parsing, formulating a response to the query string at least in part by:
querying one or more log data stores using the at least one data field, wherein the plurality of log messages are stored at the one or more log data stores,
receiving, for each of the one or more log data stores, partial result data in response to the querying of the log data store, the partial result data including one or more results of querying log messages stored at the log data store,
aggregating the partial result data received from each of the one or more log data stores to determine a preliminary result for the at least one data field, and
performing the function on the preliminary result to determine a formulated response to the query string that included the one or more virtual data fields;
generating a log field view representing the formulated response to the query string, wherein the formulated response to the query

Assignees

• Oracle Int Corp

Inventors

• Das Sreeji
• Yoon Jae Young
• Kumaresan Dhileeban
• Alvenkar Venktesh
• Akali Harish
• Wong Adrienne

Classifications

• G06F17/40 Primary

  Data acquisition and logging (for input to computer G06F3/00) · CPC title

• G06F16/2471

  Distributed queries · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F16/248 Primary

  Presentation of query results · CPC title

• G06F11/3476

  Data logging (G06F11/14, G06F11/2205 take precedence) · CPC title