Event log system

Patent metadata

Field	Value
Publication number	US-2016004733-A1
Application number	US-201314763488-A
Country	US
Kind code	A1
Filing date	Jan 31, 2013
Priority date	Jan 31, 2013
Publication date	Jan 7, 2016
Grant date	—

Abstract

An event log system and method are disclosed. The system includes a plurality of worker nodes to receive and store event log records from a plurality of log sources, and a coordinator node to maintain information about mapping from the plurality of log sources to the plurality of worker nodes. Each worker node comprises an IngestKV module to temporarily store event log records from the log sources and a TimeRangeKV module to permanently store the event log records.

First claim

What is claimed is:

1 . An event log system, comprising:
a plurality of worker nodes to receive and store event log records from a plurality of log sources; and a coordinator node to maintain information about mapping from said plurality of log sources to said plurality of worker nodes,
wherein each of said worker nodes comprises an IngestKV module to temporarily store event log records from the plurality of log sources and a TimeRangeKV module to permanently store said event log records.

2 . The event log system of claim 1 , wherein said TimeRangeKV module stores event log records based on a time range unit (TRU).

3 . The event log system of claim 2 , wherein all the event log records within the same TRU are stored on the same worker node.

4 . The event log system of claim 1 , wherein said IngestKV module is further to parse a received event log record into key-value pairs, and said TimeRangeKV module is to store the event log record in a form of timestamp and key-value pairs.

5 . The event log system of claim 2 , wherein said IngestKV module is to cache a number of event log records and then shuffle the cached event log records to said TimeRangeKV module on a corresponding worker node in rounds for storage.

6 . The event log system of claim 2 , wherein the size of said TRU is dependent upon workload.

7 . The event log system of claim 5 , wherein during shuffling, a number of worker nodes simultaneously receive shuffled event log records.

8 . The event log system of claim 7 , wherein the number of worker nodes that simultaneously receive shuffled event log records is selected at least partly based on at least one of the number of the worker nodes, network bandwidth of a single worker node, total log ingestion bandwidth and log compression ratio.

9 . The event log system of claim 1 , wherein said coordinator node is further to assign the log sources to said worker nodes.

10 . The event log system of claim 9 , wherein said coordinator node assigns the log sources to said worker nodes by:
assigning individable log sources to the worker nodes; and then assigning dividable log sources to the worker nodes to balance throughput assigned among the worker nodes.

11 . The event log system of claim 1 , wherein said IngestKV module and said TimeRangeKV module both replicate the event log records.

12 . The event log system of claim 11 , wherein each TimeRangeKV module stores the event log records within a number of adjacent TRUs.

13 . The event log system of claim 12 , wherein a windows join operation with a windows size less than or equal to twice the number of adjacent TRUs minus one can be performed locally on a worker node.

14 . A method for managing an event log, comprising:
mapping a plurality of log sources to a plurality of worker nodes; receiving by each of said plurality of worker nodes event log records and parsing them into key-value pairs; and storing the event log records in a form of timestamp and key-value pairs based on a time range unit (TRU).

15 . The method of claim 14 , wherein all the event log records within the same TRU are stored on the same worker node.

Assignees

• Cao Zhao
• Chen Shimin
• Li Feifei
• Wang Xiaoyang
• Hewlett Packard Development Co

Inventors

• Cao Zhao
• Chen Shimin
• Li Feifei
• Wang Min
• Wang Xiaoyang

Classifications

• G06F16/22 Primary

  Indexing; Data structures therefor; Storage structures · CPC title

• G06F40/205

  Parsing · CPC title

• G06F11/0709

  in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems · CPC title

• G06F21/552

  involving long-term monitoring or reporting · CPC title

• H04L67/10 Primary

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title