Retention management for data streams

Patent metadata

Field	Value
Publication number	US-11604759-B2
Application number	US-202016864892-A
Country	US
Kind code	B2
Filing date	May 1, 2020
Priority date	May 1, 2020
Publication date	Mar 14, 2023
Grant date	Mar 14, 2023

Abstract

The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

First claim

What is claimed is:

1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
receiving a request to truncate a data stream stored via a streaming data storage system, the request associated with a stream cut object maintained for the data stream, wherein the stream cut object is stored via a stream cut auxiliary stream, and wherein the stream cut object corresponds to a stream cut object position within the stream cut auxiliary stream;
obtaining a retention period of selectable retention periods associated with the data stream, wherein the retention period reflects a minimum time for which data of the data stream is to be preserved and after which the data is permitted to be deleted from the data stream;
obtaining a stream cut time associated with the stream cut object, wherein the stream cut time corresponds to an attempt to truncate the data of the data stream occurring earlier than the stream cut time;
determining whether the stream cut time is within the retention period, wherein the stream cut time being within the retention period corresponds to the attempt to truncate the data stream attempting to delete at least some of the data that is not yet permitted to be deleted from the data stream; and
in response to the stream cut time being determined not to be within the retention period, truncating the data stream based on the stream cut object.

2. The system of claim 1 , wherein the determining whether the stream cut time is within the retention period comprises determining a retention end time based on the stream cut time and the retention period, and determining whether the retention end time is later than a current system time.

3. The system of claim 1 , wherein the operations further comprise, in response to the stream cut time being determined to be within the retention period, blocking the request.

4. The system of claim 3 , wherein the request to truncate the data stream is received from a size-based expiration requestor, and wherein the blocking the request corresponds to increasing a capacity footprint of the data stream beyond a pre-specified size limit.

5. The system of claim 4 , wherein the operations further comprise scheduling a delayed truncation operation in response to the increasing the capacity footprint of the data stream.

6. The system of claim 5 , wherein the scheduling the delayed truncation operation comprises scheduling the delayed truncation operation to occur at a stream cut time that is not within the retention time period.

7. The system of claim 1 , wherein the stream cut object is a second stream cut object and the stream cut time value is a second stream cut time value that is after a first stream cut time value of a first stream cut object, stored via the stream cut auxiliary stream, earlier in time than the second stream cut object, and wherein the operations further comprise,
in response to determining that the second stream cut time value is within the retention period, selecting the first stream cut object, and determining whether the first stream cut time value is within the retention period, and
in response to determining that the first stream cut time is not within the retention period, truncating the data stream based on the first stream cut object.

8. The system of claim 1 , wherein the operations further comprise maintaining the steam cut auxiliary stream comprising the stream cut object, wherein the stream cut object further corresponds to a stream cut event creation time.

9. The system of claim 8 , wherein the operations further comprise, in response to the stream cut time being determined not to be within the retention period, truncating the auxiliary stream based on the stream cut object.

10. A method, comprising:
receiving, by a system comprising a processor, a request to truncate a data stream stored via a streaming data storage system, the request associated with a stream cut object maintained for the data stream, wherein the stream cut object is stored via a stream cut auxiliary stream, and wherein the stream cut object corresponds to a stream cut object position within the stream cut auxiliary stream;
determining, by the system, a retention period of retention periods associated with the data stream, wherein the retention period reflects a time for which data of the data stream is to be preserved and after which the data is permitted to be deleted from the data stream;
determining, by the system, a stream cut time associated with the stream cut object, wherein the stream cut time corresponds to a truncation operation to the data of the data stream occurring earlier than the stream cut time;
determining, by the system, whether the stream cut time is within the retention period, wherein the stream cut time being within the retention period corresponds to the truncation operation to the data stream attempting to delete at least some of the data that is not yet permitted to be deleted from the data stream; and
in response to the stream cut time being determined, by the system, not to be within the retention period, causing the truncation operation to the data stream to occur based on the stream cut object.

11. The method of claim 10 , wherein the determining that the stream cut time is not comprised in the retention period is based on the retention period having an end time is later than a current system time.

12. The method of claim 10 , wherein the determining that the stream cut time is not comprised in the retention period is based on the retention period having an end time is earlier than a current system time.

13. The method of claim 10 , further comprising, in response to the stream cut time being determined to be within the retention period, blocking, by the system, the request to prevent truncation of the data stream prior to the retention period.

14. The method of claim 10 , wherein the receiving the request comprises receiving the request from a size-based expiration requestor, and wherein the request corresponds to increasing a capacity footprint of the data stream beyond a pre-specified size limit.

15. The method of claim 14 , further comprising scheduling, by the system, a delayed truncation operation in response to the increasing the capacity footprint of the data stream.

16. The method of claim 15 , wherein the scheduling results in causing the delayed truncation operation to occur at a stream cut time that is not within the retention time period.

17. A machine-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
receiving a request to truncate a data stream stored via a streaming data storage system, the request associated with a stream cut object maintained for the data stream, wherein the stream cut object is stored via a stream cut auxiliary stream, and wherein the stream cut object corresponds to a stream cut object position within the stream cut auxiliary stream;
receiving a retention period of selectable retention periods associated with the data stream, wherein the retention period reflects a minimum time for which data of the data stream is to be preserved and after which the data is permitted to be deleted from the data stream;
receiving a stream cut time associated with the stream cut object, wherein the stream cut time corresponds to an attempt to truncate the data of the data stream occurring earlier than the stream cut time;
in response to determining that the stream c

Assignees

• Emc Ip Holding Co Llc

Inventors

• Danilov Mikhail
• Medvedev Igor

Classifications

• G06F16/182

  Distributed file systems · CPC title

• G06F16/125 Primary

  characterised by the use of retention policies (retention policies for HSM systems G06F16/185) · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title

• G06F16/1734

  Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs · CPC title