Centralized logging for a data storage system

Patent metadata

Field	Value
Publication number	US-9507798-B1
Application number	US-201314143993-A
Country	US
Kind code	B1
Filing date	Dec 30, 2013
Priority date	Dec 30, 2013
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

A technique for logging events in a data storage system involves designating one subset of storage processors of the data storage system as clients that generate log entries and another subset of the storage processors as servers that receive log entries. Only one server is active at a time. The active server receives the generated log entries from the clients and persists the log entries to a centralized log store. Clients assign first timestamps to the log entries based on locally accessible clocks. The active server receives the log entries, including the first timestamps, from the clients and applies second timestamps based on a clock accessible to the server. As the second timestamps are consistent across the different clients, the second timestamps can be applied to correct misalignments in time among the log entries received from the clients.

First claim

What is claimed is:

1. A method of logging events in a data storage system, the method comprising:
running a logging server on a storage processor of the data storage system;
receiving, by the logging server, log entries from multiple logging clients running on respective other storage processors of the data storage system, each log entry generated by one of the logging clients in response to an event occurring on the storage processor on which the logging client runs and including a first timestamp referenced to a clock on the storage processor on which the logging client runs;
applying, by the logging server, a second timestamp to each of the log entries received from the logging clients, each second timestamp referenced to a common clock accessible to the storage processor on which the logging server runs;
persistently storing, by the logging server, the log entries including the respective first and second timestamps to a centralized log store;
queuing the log entries in a cache of storage processors running the logging clients; and
instructing the logging clients to send the queued log entries to a standby logging server, the standby logging server being configured to apply the second timestamp to received log entries and persistently store these log entries in the centralized log store.

2. A method as in claim 1 ,
wherein each first timestamp is indicative of a time at which the respective event occurred;
wherein each second timestamp is indicative of a time at which the logging server received the respective log entry, and
wherein persistently storing each received log entry includes:
placing the log entries into a log file stored in the centralized log store, the log entries being placed in the order in which the logging server receives the log entries.

3. A method as in claim 2 , further comprising:
resequencing the log entries in the log file based on the first timestamps and the second timestamps to place the log entries in the order in which the respective events occurred.

4. A method as in claim 2 , further comprising:
running a logging gateway on a storage processor of the data storage system, the logging gateway presenting a logging server interface to a set of the logging clients and presenting a logging client interface to the logging server;
wherein receiving log entries from the logging clients includes:
obtaining at least some of the log entries from the logging gateway, the logging gateway having received such log entries from at least some of the logging clients.

5. A method as in claim 4 ,
wherein the log entries are generated in a first data format by application programs running on the storage processors on which the logging clients are run; and
wherein the method further comprises:
after receiving the log entries and prior to adding the second timestamps, translating the log entries from the first data format to a second data format.

6. A method as in claim 5 ,
wherein the first data format of the log entries is a compact binary format configured to be read by an application that runs on a storage processor running a logging client; and
wherein translating the log entries includes:
performing an unpacking operation on one or the log entries that extracts log information in the compact binary format and writes the extracted log information to a human-readable, textual format.

7. A data storage system, comprising:
a centralized log store; and
multiple storage processors, including a storage processor coupled to the centralized log store, the storage processor comprising:
a set of processing units and memory, coupled to the set of processing units, the set of processing units and the memory together realizing control circuitry, the control circuitry constructed and arranged to:
run a logging server;
receive, by the logging server, log entries from multiple logging clients running on respective other storage processors of the data storage system, each log entry generated by one of the logging clients in response to an event occurring on the storage processor on which the logging client runs and including a first timestamp referenced to a clock on the storage processor on which the logging client runs;
apply, by the logging server, a second timestamp to each of the log entries received from the logging clients, each second timestamp referenced to a common clock accessible to the storage processor on which the logging server runs; and
persistently store, by the logging server, the log entries including the respective first and second timestamps to the centralized log store;
queue the log entries in a cache of storage processors running the logging clients; and
instruct the logging clients to send the queued log entries to a standby logging server, the standby logging server being configured to apply the second timestamp to received log entries and persistently store these log entries in the centralized log store.

8. A data storage system as in claim 7 ,
wherein each first timestamp is indicative of a time at which the respective event occurred;
wherein each second timestamp is indicative of a time at which the logging server received the respective log entry, and
wherein the controlling circuitry constructed and arranged to persistently store each received log entry is further constructed and arranged to:
place the log entries into a log file stored in the centralized log store, the log entries being placed in the order in which the logging server receives the log entries.

9. A data storage system as in claim 8 , wherein the controlling circuitry is further constructed and arranged to:
sequence the log entries in the log file based on the first timestamps and the second timestamps to place the log entries in the order in which the respective events occurred.

10. A data storage system as in claim 7 , further including a set of processors configured to run logging gateways, each of the logging gateways presenting a logging server interface to a set of the logging clients and presenting a logging client interface to the logging server;
wherein the controlling circuitry constructed and arranged to receive the log entry from the processors configured as logging clients is further constructed and arranged to:
obtain at least some of the log entries from the logging gateway, the logging gateway having received such log entries from at least some of the logging clients.

11. A data storage system as in claim 7 ,
wherein the log entries are generated by application programs running on the storage processors on which the logging clients run in a first data format; and
wherein the controlling circuitry is further constructed and arranged to:
after receiving the log entries and prior to adding the second timestamps, translate the log entries from the first data format to a second data format.

12. A data storage system as in claim 11 ,
wherein the first data format of a log entry is a compact binary format configured to be read by a particular application that runs on a storage processor running a logging client; and
wherein the controlling circuitry constructed and arranged to translate the log entries is further constructed and arranged to:
perform an unpacking operation on the log entry that extracts log information from the compact binary format and writes the extracted log information to a human-readable, textual format.

13. A computer program product having a non-transitory, computer-readable storage medium which stores instructions that, when executed by a controller, cause the controller to carry out a method of logging events in a data storage system, the method compri

Assignees

• Emc Corp
• Emc Ip Holding Co Llc

Inventors

• Zhao Bingfeng
• Chen Jun
• Lathrop Stephen M
• Yim Wai Chuen
• Xie Yongmei

Classifications

• G06F11/3034

  where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097) · CPC title

• G06F16/1734 Primary

  Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs · CPC title

• G06F17/40

  Data acquisition and logging (for input to computer G06F3/00) · CPC title

• G06F2201/835

  Timestamp · CPC title

• G06F17/30194 Primary

  Physics · mapped topic