Dynamic ingestion throttling of data log

Patent metadata

Field	Value
Publication number	US-9747222-B1
Application number	US-201615086644-A
Country	US
Kind code	B1
Filing date	Mar 31, 2016
Priority date	Mar 31, 2016
Publication date	Aug 29, 2017
Grant date	Aug 29, 2017

Abstract

A technique for controlling acceptance of host application data into a data log in a data storage system includes selectively accepting or refusing newly arriving host data into the data log based on a comparison between an oldest entry in the data log and an age threshold. The age threshold is dynamically updated based on system heuristics. As long as the oldest log entry is younger than the age threshold, the data log continues to accept newly arriving host application data, acknowledging IO requests to host applications as the data specified in those requests is entered into the log. However, when the oldest log entry is older than the age threshold, new log entries are temporarily refused entry into the data log. Instead, newly arriving data are placed in a pending list, where they are kept until the data log is again accepting new log entries.

First claim

What is claimed is:

1. A method of dynamically controlling acceptance of host application data into a data log in a data storage system, the method comprising:
generating a throttle signal, the throttle signal having (i) a first value when an oldest log entry in the data log is younger than an age threshold and (ii) a second value when the oldest log entry in the data log is older than the age threshold, the data storage system dynamically updating the age threshold based on a set of heuristics pertaining to the data storage system;
in response to the throttle signal having the first value and the data storage system receiving a first IO request from a host application, (i) storing data specified in the first IO request in a first log entry of the data log and (ii) acknowledging completion of the first IO (Input/Output) request to the host application;
in response to the throttle signal having the second value and the data storage system receiving a second IO request from the host application, storing data specified in the second IO request in a pending list; and
after storing the data specified in the second IO request in the pending list and in response to the throttle signal then assuming the first value, (i) storing the data specified in the second IO request in a second log entry of the data log and (ii) acknowledging completion of the second IO request to the host application.

2. The method of claim 1 , further comprising:
with the throttle signal having the second value, flushing multiple log entries from the data log, the log entries including the oldest log entry, such that a log entry younger than the oldest log entry becomes a new oldest log entry; and
changing the throttle signal from the second value to the first value in response to the new oldest log entry in the data log being older than the age threshold.

3. The method of claim 2 , wherein the data storage system persistently stores the data log but non-persistently stores the pending list.

4. The method of claim 3 , wherein the set of heuristics includes a measure of fullness of the data log, and wherein the method further comprises:
decreasing the age threshold in response to detecting that the measure of fullness of the data log has increased; and
increasing the age threshold in response to detecting that the measure of fullness of the data log has decreased.

5. The method of claim 3 , wherein the set of heuristics includes an age of a current oldest entry in the data log, and wherein the method further comprises:
decreasing the age threshold in response to detecting that the age of the current oldest entry in the data log has increased; and
increasing the age threshold in response to detecting that the age of the current oldest entry in the data log has decreased.

6. The method of claim 3 , further comprising, when storing data in log entries of the data log, also storing metadata describing those log entries in a volatile data cache, wherein the set of heuristics includes a measure of fullness of the volatile data cache, and wherein the method further comprises:
decreasing the age threshold in response to detecting that the measure of fullness of the volatile data cache has increased; and
increasing the age threshold in response to detecting that the measure of fullness of the volatile data cache has decreased.

7. The method of claim 3 , further comprising:
when storing data in log entries of the data log, also storing metadata describing those log entries in a volatile data cache,
wherein the set of heuristics includes (i) a measure of fullness of the data log, (ii) an age of a current oldest entry in the data log, and (iii) a measure of fullness of the volatile data cache,
and wherein the method further comprises:
decreasing the age threshold in response to detecting an increase in any of (i) the measure of fullness of the data log, (ii) the age of the current oldest entry in the data log, and (iii) the measure of fullness of the volatile data cache; and
increasing the age threshold in response to detecting a decrease in any of (i) the measure of fullness of the data log, (ii) the age the current oldest entry in the data log, and (iii) the measure of fullness of the volatile data cache.

8. The method of claim 7 , wherein dynamically updating the age threshold based on a set of heuristics includes regenerating the age threshold on a periodic basis.

9. The method of claim 8 , wherein the method further comprises decreasing the age threshold by at least hundreds of milliseconds in response to the data log becoming full.

10. The method of claim 9 , wherein the method further comprises decreasing the age threshold by at least hundreds of milliseconds in response to the data cache becoming full.

11. The method of claim 10 , wherein the data storage system includes a data log manager that manages the data log, and wherein the method further comprises:
backing a set of log entries in the data log in a cache structure, the cache structure configured to flush each of the set of log entries to non-volatile storage either (i) to a first storage location or (ii) to a second storage location;
receiving, by the data log manager, an alert from the cache structure, the alert indicating that at least one of the set of log entries has been flushed to the first storage location; and
in response to receiving the alert, decreasing the age threshold by at least hundreds of milliseconds.

12. A data storage system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:
generate a throttle signal, the throttle signal having (i) a first value when an oldest log entry in the data log is younger than an age threshold and (ii) a second value when the oldest log entry in the data log is older than the age threshold, the data storage system configured to dynamically update the age threshold based on a set of heuristics pertaining to the data storage system;
in response to the throttle signal having the first value and the data storage system receiving a first IO request from a host application, (i) store data specified in the first IO (Input/Output) request in a first log entry of the data log and (ii) acknowledge completion of the first IO request to the host application;
in response to the throttle signal having the second value and the data storage system receiving a second IO request from the host application, store data specified in the second IO request in a pending list; and
after storing the data specified in the second IO request in the pending list and in response to the throttle signal then assuming the first value, (i) store the data specified in the second IO request in a second log entry of the data log and (ii) acknowledge completion of the second IO request to the host application.

13. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a data storage system, cause the control circuitry to perform a method for dynamically controlling acceptance of host application data into a data log in a data storage system, the method comprising:
generating a throttle signal, the throttle signal having (i) a first value when an oldest log entry in the data log is younger than an age threshold and (ii) a second value when the oldest log entry in the data log is older than the age threshold, the data storage system dynamically updating the age threshold based on a set of heuristics pertaining to the data storage system;
in response to the throttle signal having the first value and the data storage system rece

Assignees

• Emc Ip Holding Co Llc

Inventors

• Armangau Philippe
• Caram Bruce E
• Seibel Christopher A

Classifications

• G06F13/1642

  with request queuing · CPC title

• G06F12/124 Primary

  being minimized, e.g. non MRU · CPC title

• G06F2212/312

  In storage controller · CPC title

• G06F2212/314

  In storage network, e.g. network attached cache · CPC title

• G06F13/4234

  being a memory bus · CPC title