Event log system

Patent metadata

Field	Value
Publication number	US-10019471-B2
Application number	US-201314763488-A
Country	US
Kind code	B2
Filing date	Jan 31, 2013
Priority date	Jan 31, 2013
Publication date	Jul 10, 2018
Grant date	Jul 10, 2018

Abstract

An event log system and method are disclosed. The system includes a plurality of worker nodes to receive and store event log records from a plurality of log sources, and a coordinator node to maintain information about mapping from the plurality of log sources to the plurality of worker nodes. Each worker node comprises an IngestKV module to temporarily store event log records from the log sources and a TimeRangeKV module to permanently store the event log records.

First claim

What is claimed is:

1. An event log system, comprising:
a plurality of worker nodes to receive and store event log records from a plurality of log sources;
a coordinator node to maintain information about mapping from said plurality of log sources to said plurality of worker nodes, wherein each of said worker nodes comprises a processor to cause:
an IngestKV module to temporarily store event log records from the plurality of log sources; and
a TimeRangeKV module to permanently store said event log records within a plurality of adjacent time range units (TRUs), wherein the IngestKV and the TimeRangeKV perform a window join operation and wherein the IngestKV module and the TimeRangeKV module replicate said event log records.

2. The event log system of claim 1 , wherein said TimeRangeKV module stores event log records based on the (TRU).

3. The event log system of claim 2 , wherein all the event log records within the same TRU are stored on the same worker node.

4. The event log system of claim 2 , wherein said IngestKV module is to cache a number of event log records and then shuffle the cached event log records to said TimeRangeKV module on a corresponding worker node in rounds for storage.

5. The event log system of claim 4 , wherein during shuffling, a number of worker nodes simultaneously receive shuffled event log records.

6. The event log system of claim 5 , wherein the number of worker nodes that simultaneously receive shuffled event log records is selected at least partly based on at least one of the number of the worker nodes, network bandwidth of a single worker node, total log ingestion bandwidth and log compression ratio.

7. The event log system of claim 2 , wherein the size of said TRU is dependent upon workload.

8. The event log system of claim 1 , wherein said IngestKV module is further to parse a received event log record into key-value pairs, and said TimeRangeKV module is to store the event log record in a form of timestamp and key-value pairs.

9. The event log system of claim 1 , wherein said coordinator node is further to assign the log sources to said worker nodes.

10. The event log system of claim 9 , wherein said coordinator node assigns the log sources to said worker nodes by:
assigning individable log sources to the worker nodes; and
then assigning dividable log sources to the worker nodes to balance throughput assigned among the worker nodes.

11. The event log system of claim 1 , wherein the windows join operation with a windows size less than twice the size of one TRU.

12. A method for managing an event log, comprising:
mapping a plurality of log sources to a plurality of worker nodes, wherein each of said worker nodes comprises a processor to cause:
an IngestKV module to temporarily store event log records from the plurality of log sources; and
a TimeRangeKV module to permanently store said event log records within a plurality of adjacent time range units (TRUs), wherein the IngestKV and the TimeRangeKV perform a window join operation and wherein the IngestKV module and TimeRangeKV module replicate said event log records; and

receiving by each of said plurality of worker nodes event log records and parsing them into key-value pairs; and
storing the event log records in a form of timestamp and key-value pairs based on the (TRU).

13. The method of claim 12 , wherein all the event log records within the same TRU are stored on the same worker node.

Assignees

• Hewlett Packard Development Co
• Cao Zhao
• Chen Shimin
• Li Feifei
• Wang Min
• Wang Xiaoyang
• Hewlett Packard Entpr Dev Lp

Inventors

• Cao Zhao
• Chen Shimin
• Li Feifei
• Wang Min
• Wang Xiaoyang

Classifications

• G06F11/0709

  in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems · CPC title

• G06F40/205

  Parsing · CPC title

• G06F11/0778

  Dumping, i.e. gathering error/state information after a fault for later diagnosis · CPC title

• G06F21/552

  involving long-term monitoring or reporting · CPC title

• H04L67/10 Primary

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title