Demoted data retention via a tiered ordered event stream data storage system

Patent metadata

Field	Value
Publication number	US-11971850-B2
Application number	US-202117503024-A
Country	US
Kind code	B2
Filing date	Oct 15, 2021
Priority date	Oct 15, 2021
Publication date	Apr 30, 2024
Grant date	Apr 30, 2024

Abstract

Data retention for a tiered ordered event stream (OES) data storage system is disclosed. Data retention can be applied to demoted data, e.g., data demoted from a first tier to a second tier of data storage devices. The data retention disclosed can avoid demoted events, generally aggregated via a chunk object, file object, etc., from being subject to both a stream-cut operation and a retention period. In embodiments, where a stream-cut instance corresponds to events occurring later in time than a retention end time (RET), all demoted events, e.g., as aggregated by a chunk, file, etc., can be affiliated with the RET. However, where the stream-cut instance corresponds to events occurring earlier in time than the RET, then a portion of the events, e.g., those that will be truncated by performing the stream-cut instance, can be demoted and affiliated with the RET.

First claim

What is claimed is:

1. A device, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
determining a first group of events between a first time and a second time of an ordered event stream to be demoted from a first tier of data storage devices of a data storage system to storage in a chunk in a second tier of data storage devices of the data storage system, wherein the first tier of data storage devices employ a stream based storage technology, and the second tier of data storage devices employ a chunk based storage technology;
determining a event retention time for the first group of events;
determining a stream-cut instance of the ordered event stream;
in response to determining that a third time of the stream-cut instance does not fall between the first time and the second time of the ordered event stream, facilitating demotion of the first group of events from the first tier of data storage devices to storage in the chunk in the second tier of data storage devices, wherein the demotion of the first group of events comprises affiliating the event retention time with the chunk, and wherein the first group of events are not permitted to be removed from the ordered event stream prior to a current time transitioning the event retention time; and
in response to determining that the third time of the stream-cut instance falls between the first time and the second time of the ordered event stream:
facilitating demotion of a first sub-group of events of the first group of events between the first time and the third time from the first tier of data storage devices to storage in a first chunk in the second tier of data storage devices, wherein the demotion of the first sub-group of events comprises affiliating the event retention time with the first chunk, and wherein the first chunk is only partially filled, and
facilitating demotion of a second sub-group of events of the first group of events between the third time and the second time from the first tier of data storage devices to storage in a second chunk in the second tier of data storage devices, and demotion of a second group of events between the second time and a fourth time of the ordered event stream from the first tier of data storage devices to storage in the second chunk.

2. The device of claim 1 , wherein the event retention time is based on an event creation time and a selectable event retention period, and wherein the selectable event retention period corresponds to an amount of time that an event of the first group of events is to be retained before permitting removal of the event from the ordered event stream.

3. The device of claim 2 , wherein the event retention period is a different period than a first-tier period that corresponds to an amount of time that the first event resided in the first tier of data storage devices before being demoted.

4. The device of claim 2 , wherein the first group of events of the ordered event stream is selected to be demoted based on a count of total events stored via the first tier of data storage devices.

5. The device of claim 2 , wherein the first group of events of the ordered event stream is selected to be demoted based on an age of an oldest event stored via the first tier of data storage devices.

6. The device of claim 1 , wherein the event retention time is a first event retention time, and wherein the operations further comprise determining a second event retention time to affiliate with the second chunk.

7. The device of claim 1 , wherein the second tier of storage devices comprises a plurality of storage devices.

8. The device of claim 7 , wherein a first storage device of the plurality of storage devices is located remotely from a second storage device of the plurality of storage devices.

9. A method, comprising:
identifying, by a system comprising a processor, a first group of events between a first time and a second time of an ordered event stream to be demoted from a first tier of data storage devices of a data storage system to storage in a chunk in a second tier of data storage devices of the data storage system, wherein the first tier of data storage devices employ a stream based storage technology, and the second tier of data storage devices employ a chunk based storage technology;
determining, by the system, an event retention time for the first group of events;
determining, by the system, a stream-cut instance of the ordered event stream;
in response to determining that a third time of the stream-cut instance does not fall between the first time and the second time of the ordered event stream: moving, by the system, the group of events from the first tier of data storage devices to storage in the chunk in the second tier of data storage devices, and affiliating the event retention time with the chunk, wherein the first group of events is not permitted to be removed from the ordered event stream prior to a current time passing the event retention time; and
in response to determining that the third time of the stream-cut instance falls between the first time and the second time of the ordered event stream:
moving, by the system, a first sub-group of events of the first group of events between the first time and the third time from the first tier of data storage devices to storage in a first chunk in the second tier of data storage devices, and affiliating, by the system, the event retention time with the first chunk, wherein the first chunk is only partially filled, and
moving, by the system, a second sub-group of events of the first group of events between the third time and the second time from the first tier of data storage devices to storage in a second chunk in the second tier of data storage devices, and moving, by the system, a second group of events between the second time and a fourth time of the ordered event stream from the first tier of data storage devices to storage in the second chunk.

10. The method of claim 9 , wherein the second tier of storage devices comprises a plurality of geographically diverse storage devices.

11. The method of claim 9 , wherein the event retention time is a first event retention time, and further comprising determining, by the system, a second event retention time to affiliate with the second chunk.

12. The method of claim 9 , wherein the event retention time is based on an event creation time and a selectable event retention period, and wherein the selectable event retention period corresponds to an amount of time that an event of the first group of events is to be retained before permitting removal of the event from the ordered event stream.

13. The method of claim 9 , wherein the first group of events of the ordered event stream are selected to be demoted based on a count of total events stored via the first tier of data storage devices.

14. The method of claim 9 , wherein the first group of events of the ordered event stream are selected to be demoted based on an age of an oldest event stored via the first tier of data storage devices.

15. A non-transitory machine-readable storage medium, comprising executable instructions that, when executed by a processor of network device of a network provider, facilitate performance of operations, comprising:
determining an event retention time for a first group of events between a first time and a second time of an ordered event stream that are selected for demotion from a first tier of data storage devices of a data storage system to storage in an object in a second tier of data storage devices of the data

Assignees

• Emc Ip Holding Co Llc

Inventors

• Danilov Mikhail
• Altaye Yohannes

Classifications

• G06F16/125 Primary

  characterised by the use of retention policies (retention policies for HSM systems G06F16/185) · CPC title

• G06F16/1805

  Append-only file systems, e.g. using logs or journals to store data · CPC title

• G06F16/185

  Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof (details of archiving G06F16/11) · CPC title