Retention management for data streams

Patent metadata

Field	Value
Publication number	US-11960441-B2
Application number	US-202318164744-A
Country	US
Kind code	B2
Filing date	Feb 6, 2023
Priority date	May 1, 2020
Publication date	Apr 16, 2024
Grant date	Apr 16, 2024

Abstract

The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

First claim

What is claimed is:

1. A method, comprising:
determining, by a streaming data storage system comprising a processor, whether a request to truncate data stream events in a data stream that are prior to a specified stream cut stored in an auxiliary stream associated with the data stream is allowable, wherein timing in the auxiliary stream and timing in the data stream are based on a same system timing protocol, and the determining comprising:
obtaining a retention period associated with the data stream;
obtaining a stream cut time associated with the specified stream cut;
evaluating the stream cut time relative to the retention period to determine that the stream cut time is within the retention period or that the stream cut time is not within the retention period;
in response to determining that the stream cut time is within the retention period;
determining that the request to truncate the data stream events is not allowable, and
blocking the request, wherein the request to truncate the data stream events is received from a size-based expiration requestor. and the blocking of the request corresponds to increasing a capacity footprint of the data stream beyond a pre-specified size limit; and

in response to determining that the stream cut time is not within the retention period;
determining that the request to truncate the data stream events is allowable, and
truncating the data stream based on the stream cut time in response to the request, wherein truncating the data stream comprises deleting the data stream events from the streaming data storage system.

2. The method of claim 1 , further comprising:
in response to the determining that the request to truncate the data stream events is not allowable, scheduling a future truncation operation based on the specified stream cut.

3. The method of claim 2 , wherein the scheduling the future truncation operation comprises scheduling a delayed truncation operation to occur at a future stream cut time that is after the stream cut time and is not within the retention period.

4. The method of claim 1 , wherein the evaluating the stream cut time relative to the retention period to determine that the stream cut time is within the retention period or that the stream cut time is not within the retention period comprises determining a retention period end time based on the stream cut time and the retention period, and determining whether the retention period end time is later than a current system time.

5. The method of claim 1 , wherein the evaluating the stream cut time relative to the retention period to determine that the stream cut time is within the retention period or that the stream cut time is not within the retention period comprises determining a retention period start time based on a current system time and the retention period, and determining whether the stream cut time is earlier than the retention period start time.

6. The method of claim 1 , further comprising:
in response to determining that the stream cut time is not within the retention period, truncating the auxiliary stream based on the stream cut time.

7. The method of claim 1 , wherein the retention period is based on a retention policy associated with the streaming data storage system.

8. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
determining whether a request to truncate data stream events is allowable, the data stream events being represented in a data stream prior to a specified stream cut stored in an auxiliary stream associated with the data stream, wherein timing in the auxiliary stream and timing in the data stream are based on a same system timing protocol, and the determining comprising,
obtaining a retention period associated with the data stream;
obtaining a stream cut time associated with the specified stream cut;
evaluating the stream cut time relative to the retention period to determine that the stream cut time is within the retention period or that the stream cut time is not within the retention period;
in response to determining that the stream cut time is within the retention period;
determining that the request to truncate the data stream events is not allowable, and
blocking the request, wherein the request to truncate the data stream events is received from a size-based expiration requestor. and the blocking of the request corresponds to increasing a capacity footprint of the data stream beyond a pre-specified size limit; and

in response to determining that the stream cut time is not within the retention period:
determining that the request to truncate the data stream events is allowable, and
truncating the data stream based on the stream cut time in response to the request, wherein truncating the data stream comprises deleting the data stream events from the streaming data storage system.

9. The system of claim 8 , wherein the operations further comprise:
in response to the determining that the request to truncate the data stream events is not allowable, scheduling a future truncation operation based on the specified stream cut.

10. The system of claim 9 , wherein the scheduling the future truncation operation comprises scheduling a delayed truncation operation to occur at a future stream cut time that is after the stream cut time and is not within the retention period.

11. The system of claim 8 , wherein the evaluating the stream cut time relative to the retention period to determine whether the stream cut time is within the retention period comprises determining a retention period end time based on the stream cut time and the retention period, and determining whether the retention period end time is later than a current system time.

12. The system of claim 8 , wherein the evaluating the stream cut time relative to the retention period to determine whether the stream cut time is within the retention period comprises determining a retention period start time based on a current system time and the retention period, and determining whether the stream cut time is earlier than the retention period start time.

13. The system of claim 8 , wherein the operations further comprise:
in response to determining that the stream cut time is not within the retention period, truncating the auxiliary stream based on the stream cut time.

14. The system of claim 8 , wherein the retention period is based on a retention policy associated with the data stream.

15. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
determining whether a request to truncate data stream events in a data stream, which are prior to a specified stream cut stored in an auxiliary stream associated with the data stream is allowable, wherein timing in the auxiliary stream and timing in the data stream are based on a same system timing protocol, and the determining comprising,
obtaining a retention period associated with the data stream;
obtaining a stream cut time associated with the specified stream cut;
evaluating the stream cut time relative to the retention period to determine that the stream cut time is within the retention period or that the stream cut time is not within the retention period;
in response to determining that the stream cut time is within the retention period;

determining that the request to truncate the data stream events is not allowable, and
blocking the request, wherein the request to truncate the data stream even

Assignees

• Emc Ip Holding Co Llc

Inventors

• Danilov Mikhail
• Medvedev Igor

Classifications

• G06F16/125 Primary

  characterised by the use of retention policies (retention policies for HSM systems G06F16/185) · CPC title

• G06F16/1734

  Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F16/24568

  Data stream processing; Continuous queries · CPC title