Linking related events for various devices and services in computer log files on a centralized server

Patent metadata

Field	Value
Publication number	US-10698756-B1
Application number	US-201816129626-A
Country	US
Kind code	B1
Filing date	Sep 12, 2018
Priority date	Dec 15, 2017
Publication date	Jun 30, 2020
Grant date	Jun 30, 2020

Abstract

A system with an interactive user interface for users to view and interact with sanitized log data received from a plurality of hosts, such as those associated with various services of an organization. The system may receive from hosts log files and/or metadata that have been filtered by agents executing on the respective hosts to remove or anonymize any sensitive or confidential information prior to transmission to the system. In some embodiments the system does further filtering of the sanitized data. Received sanitized data is parsed, indexed, and/or otherwise processed for optimal searching, and stored in a log pipeline. The system causes display of an electronic visualization interface comprising a dynamic electronic search configured to receive an indication of various log search criteria, such as an error or trace identifier, that are used to identify matching log files meeting the provided criteria, such as log files associated with services executed on different hosts.

First claim

What is claimed is:

1. A computing system for dynamically generating a dynamic electronic search system, the system comprising:
one or more computer readable storage devices configured to store a plurality of computer executable instructions; and
one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause the computing system to:
electronically receive, via one or more networks, sanitized data from a plurality of remote hosts, wherein the sanitized data comprises a plurality of log files each including unique identifiers of associated software services, wherein the remote hosts are configured to apply rules to determine data that is uploaded to a log pipeline, wherein at least some unique identifiers of software services are included in multiple log files associated with related events performed by each of multiple software services;
store, in a log pipeline comprising one or more databases, the plurality of log files;
parsing the log files into log attributes, the log attributes including at least time stamps and unique identifiers of associated software services of the respective log files;
indexing the log files, including the log attributes, into an indexed searching platform; and
cause display of an electronic visualization interface, the electronic visualization interface comprising a dynamic electronic search configured to receive an indication of a unique identifier and identifying, based on the indexed searching platform, two or more matching log files having the same unique identifier from different hosts.

2. The computing system of claim 1 , wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:
remove from the parsed log files information deemed to be sensitive.

3. The computing system of claim 1 , wherein the sanitized data includes log data and metadata from one or more of the remote hosts.

4. The computing system of claim 3 , wherein the metadata comprises at least a hostname of the host.

5. The computing system of claim 3 , wherein the metadata comprises at least an IP address.

6. The computing system of claim 1 , wherein each of the remote hosts are configured to format the sanitized data into a normalized format.

7. The computing system of claim 1 , wherein the indexed attributes comprise one or more of log freshness, error identifiers, trace identifiers, service identifiers, originating service, originating host, originating log, service version, error priority level, and magnitude of the error.

8. The computing system of claim 1 , wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable modifications to the display of the sanitized data.

9. The computing system of claim 1 , wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable customizations to the viewing pane comprising adding columns, removing columns, adding viewable data, and removing viewable data.

10. The computing system of claim 1 , wherein the same unique identifier is a particular error instance ID.

11. A method performed by at least one electronic device comprising one or more processors, the method comprising:
electronically receive, via one or more networks, sanitized data from a plurality of remote hosts, wherein the sanitized data comprises a plurality of log files each including unique identifiers of associated software services, wherein the remote hosts are configured to apply rules to determine data that is uploaded to a log pipeline, wherein at least some unique identifiers of software services are included in multiple log files associated with related events performed by each of multiple software services;
store, in a log pipeline comprising one or more databases, the plurality of log files;
parsing the log files into log attributes, the log attributes including at least time stamps and unique identifiers of associated software services of the respective log files;
indexing the log files, including the log attributes, into an indexed searching platform; and
cause display of an electronic visualization interface, the electronic visualization interface comprising a dynamic electronic search configured to receive an indication of a unique identifier and identifying, based on the indexed searching platform, two or more matching log files having the same unique identifier from different hosts.

12. The method of claim 11 , wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:
remove from the parsed log files information deemed to be sensitive.

13. The method of claim 11 , wherein the sanitized data includes log data and metadata from one or more of the remote hosts.

14. The method of claim 11 , wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable modifications to the display of the sanitized data.

15. The method of claim 11 , wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable customizations to the viewing pane comprising adding columns, removing columns, adding viewable data, and removing viewable data.

16. The method of claim 11 , wherein the unique identifier and same unique identifier is a particular error instance ID.

17. A non-transitory computer-readable medium storing a set of instructions that are executable by one or more electronic devices, each having one or more processors, to cause the one or more electronic devices to perform a method, the method comprising:
electronically receive, via one or more networks, sanitized data from a plurality of remote hosts, wherein the sanitized data comprises a plurality of log files each including unique identifiers of associated software services, wherein the remote hosts are configured to apply rules to determine data that is uploaded to a log pipeline, wherein at least some unique identifiers of software services are included in multiple log files associated with related events performed by each of multiple software services;
store, in a log pipeline comprising one or more databases, the plurality of log files;
parsing the log files into log attributes, the log attributes including at least time stamps and unique identifiers of associated software services of the respective log files;
indexing the log files, including the log attributes, into an indexed searching platform; and
cause display of an electronic visualization interface, the electronic visualization interface comprising a dynamic electronic search configured to receive an indication of a unique identifier and identifying, based on the indexed searching platform, two or more matching log files having the same unique identifier from different hosts.

18. The computer-readable medium of claim 17 , wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:
remove from the parsed log files information deemed to be sensitive.

19. The computer-readable medium of claim 17 , wherein the sanitized data includes log data and metadata from one or more of the remote hosts.

20. The computer-readable medi

Assignees

• Palantir Technologies Inc

Inventors

• Abdelsalam Mahmoud
• Santos Eric
• Dearment Greg
• Wu Grant
• Ross James
• Simons Kevin
• Xu Guodong
• Wang Peter
• Niemi Richard
• Lakshmanan Vivek

Classifications

• G06F21/6254 Primary

  by anonymising data, e.g. decorrelating personal data from the owner's identification · CPC title

• G06F17/40

  Data acquisition and logging (for input to computer G06F3/00) · CPC title

• G06F11/366

  using diagnostics (G06F11/0703 takes precedence) · CPC title

• G06F11/3006

  where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50) · CPC title

• G06F11/079

  Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36) · CPC title