Linking related events for various devices and services in computer log files on a centralized server

Patent metadata

Field	Value
Publication number	US-11442802-B2
Application number	US-202016878928-A
Country	US
Kind code	B2
Filing date	May 20, 2020
Priority date	Dec 15, 2017
Publication date	Sep 13, 2022
Grant date	Sep 13, 2022

Abstract

A system with an interactive user interface for users to view and interact with sanitized log data received from a plurality of hosts, such as those associated with various services of an organization. The system may receive from hosts log files and/or metadata that have been filtered by agents executing on the respective hosts to remove or anonymize any sensitive or confidential information prior to transmission to the system. In some embodiments the system does further filtering of the sanitized data. Received sanitized data is parsed, indexed, and/or otherwise processed for optimal searching, and stored in a log pipeline. The system causes display of an electronic visualization interface comprising a dynamic electronic search configured to receive an indication of various log search criteria, such as an error or trace identifier, that are used to identify matching log files meeting the provided criteria, such as log files associated with services executed on different hosts.

First claim

What is claimed is:

1. A computing system comprising:
one or more computer readable storage devices configured to store a plurality of computer executable instructions; and
one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause the computing system to:
generate electronic instructions comprising logging specifications for generating sanitized log files that are based at least in part on one or more log files associated with events performed by each of multiple software services executed by each of a plurality of remote hosts;
transmit, to each of the plurality of remote hosts by one or more networks, the electronic instructions, wherein the logging specifications include configurable rules for each of the plurality of remote hosts to automatically execute to determine data that is uploaded to a log pipeline so that the sanitized log files generated by each of the plurality of remote hosts include a subset of the data included in the one or more log files such that sensitive user data captured by each of the plurality of remote hosts is removed from the one or more log files by the plurality of remote hosts while generating the sanitized log files, and wherein the subset of the data included in the one or more log files is further generated to be in a uniform data format that is in compliance with the configurable rules;
electronically receive, from each of the plurality of remote hosts by one or more networks, the sanitized log files;
index the sanitized log files into an indexed searching platform, wherein the indexing is based at least in part on the sanitized log files being in the uniform data format that is in compliance with the configurable rules; and
generate instructions to cause display of an electronic visualization interface comprising a dynamic electronic search configured to receive search criteria, wherein the electronic visualization interface is configured to identify, based on the indexed searching platform and the search criteria, any two or more sanitized log files from different hosts that satisfy the search criteria.

2. The computing system of claim 1 , wherein the two or more sanitized log files are determined based on unique identifiers associated with each of the log files such that the two or more sanitized log files have the same unique identifier.

3. The computing system of claim 2 , wherein each of the unique identifiers indicates software services of the respective log files.

4. The computing system of claim 2 , wherein the same unique identifier is a particular error instance ID.

5. The computing system of claim 1 , wherein each log file of the log files includes log data and metadata associated with a respective remote host of the plurality of remote hosts.

6. The computing system of claim 1 , wherein the indexed sanitized log files comprise one or more of the following: log freshness, error identifiers, trace identifiers, service identifiers, originating service, originating host, originating log, service version, error priority level, and magnitude of the error.

7. The computing system of claim 1 , wherein the electronic visualization interface further comprises a customizable viewing pane configured to receive modifications to the display of the log files or information associated with the log files.

8. The computing system of claim 1 , wherein the electronic visualization interface further comprises a customizable viewing pane configured to receive customizations to the customizable viewing pane comprising adding columns, removing columns, adding viewable data, or removing viewable data.

9. The computing system of claim 1 , wherein the plurality of remote hosts includes a first remote host and a second remote host, wherein the first remote host is associated with first configurable rules and the second remote host is associated with second configurable rules, and wherein the first configurable rules and the second configurable rules are different.

10. The computing system of claim 1 , wherein the sensitive user data associated with each of each of the plurality of remote hosts originates on each respective remote host such that the sensitive user data is captured or created on the respective remote host.

11. The computing system of claim 1 , wherein the sensitive user data is removed from the one or more log files by each of the plurality of remote hosts but that the sensitive user data otherwise remains stored on each of the plurality of remote hosts.

12. The computing system of claim 1 , wherein the electronic visualization interface is further configured to include one or more visualizations based at least in part on the sanitized log files, wherein the one or more visualizations comprise metrics associated with a quantity of occurrences of one or more log entries in the sanitized log files.

13. The computing system of claim 1 , wherein the one or more hardware computer processors are further configured to: generate a plurality of unique alphanumeric identifiers that correspond to one or more of the multiple software services running on each of the plurality of remote hosts, wherein the plurality of unique alphanumeric identifiers are generated based at least in part on the configurable rules, and wherein the identified any two or more sanitized log files are associated with the same unique identifier.

14. A method performed by at least one electronic device comprising one or more processors, the method comprising:
generating electronic instructions comprising logging specifications for generating sanitized log files that are based at least in part on one or more log files associated with events performed by each of multiple software services executed by each of a plurality of remote hosts;
transmitting, to each of the plurality of remote hosts by one or more networks, the electronic instructions, wherein the logging specifications include configurable rules for each of the plurality of remote hosts to automatically execute to determine data that is uploaded to a log pipeline so that the sanitized log files generated by each of the plurality of remote hosts include a subset of the data included in the one or more log files such that sensitive user data captured by each of the plurality of remote hosts is removed from the one or more log files by the plurality of remote hosts while generating the sanitized log files, and wherein the subset of the data included in the one or more log files is further generated to be in a uniform data format that is in compliance with the configurable rules;
electronically receiving, from each of the plurality of remote hosts by one or more networks, the sanitized log files;
indexing the sanitized log files into an indexed searching platform, wherein the indexing is based at least in part on the sanitized log files being in the uniform data format that is in compliance with the configurable rules; and
generating instructions to cause display of an electronic visualization interface comprising a dynamic electronic search configured to receive search criteria, wherein the electronic visualization interface is configured to identify, based on the indexed searching platform and the search criteria, any two or more sanitized log files from different hosts that satisfy the search criteria.

15. The method of claim 14 , wherein the two or more sanitized log files are determined based on unique identifiers associated with each of the log files such that the two or more sanitized log files have the same unique id

Assignees

• Palantir Technologies Inc

Inventors

• Abdelsalam Mahmoud
• Santos Eric
• Dearment Greg
• Wu Grant
• Ross James
• Simons Kevin
• Xu Guodong
• Wang Peter
• Niemi Richard
• Lakshmanan Vivek

Classifications

• G06F11/3072

  where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting · CPC title

• G06F11/3476

  Data logging (G06F11/14, G06F11/2205 take precedence) · CPC title

• G06F11/0787

  Storage of error reports, e.g. persistent data storage, storage using memory protection · CPC title

• G06F11/079

  Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36) · CPC title

• G06F11/0775

  Content or structure details of the error report, e.g. specific table structure, specific error fields · CPC title