Ordered event stream event retention

Patent metadata

Field	Value
Publication number	US-11599420-B2
Application number	US-202016944094-A
Country	US
Kind code	B2
Filing date	Jul 30, 2020
Priority date	Jul 30, 2020
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

Retention of events of an ordered event stream is disclosed. Expiration of events stored in a segment of an ordered event stream (OES) can be desirable. New events are added to a head of an OES segment, and pruning events from a tail of the OES segment can be valuable. Processing applications can register a processing scheme for a segment, e.g., at-least-once processing, exactly-once processing, etc., and can generate checkpoints indicating a degree of advancement in processing events of the segment. The ordered event stream can determine a cut point indicative of a progress point, that before which, events of an OES can be marked as ready for expiration. However, events that are marked for expiration can be retained to allow processing based on a checkpoint, e.g., expiration of the event can be refused until there is an assurance the event was read by the processing application.

First claim

What is claimed is:

1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
receiving a first application processor checkpoint corresponding to a first segment of an ordered event stream, wherein the ordered event stream is stored via an ordered event stream storage system;
receiving an event expiration point corresponding to the ordered event stream, and
in response to determining an expiration status of an event stored via the first segment of the ordered event stream based on the first application processor checkpoint and the event expiration point, causing retention of the event in the ordered event stream.

2. The system of claim 1 , wherein:
the expiration status indicates that the event expiration point occurs at an earlier progress point than the first application processor checkpoint; and
the facilitating the retention of the event results in prohibiting deletion of the event.

3. The system of claim 1 , wherein:
the expiration status indicates that the event expiration point occurs at a later progress point than the first application processor checkpoint; and
the facilitating the retention of the event results in permitting deletion of the event.

4. The system of claim 1 , wherein the operations further comprise:
determining that the first application processor checkpoint is an earliest checkpoint of checkpoints comprising the first application processor checkpoint and a second application processor checkpoint corresponding to a second segment of the ordered event stream.

5. The system of claim 4 , wherein the second segment is a same segment as the first segment.

6. The system of claim 4 , wherein the second segment is a different segment that the first segment.

7. The system of claim 6 , wherein the first segment is in a first epoch that is a different epoch than a second epoch comprising the second segment.

8. The system of claim 7 , wherein the determining that the first application processor checkpoint is an earliest checkpoint comprises:
determining that the first segment is comprised in the first epoch;
determining that the second segment is comprised in the second epoch;
determining that the first epoch occurs at an earlier progress point than the second epoch; and
determining that the first application processor checkpoint is an earlier application processor checkpoint than any other application processor checkpoints comprised in any segments of the OES comprised in the first epoch.

9. The system of claim 7 , wherein first epoch and the second epoch are a result of scaling the ordered event stream.

10. The system of claim 4 , wherein the determining that the first application processor checkpoint is an earliest checkpoint of checkpoints comprising the first application processor checkpoint and a second application processor checkpoint corresponding to a second segment of the ordered event stream results in facilitating retention of another event stored via the second segment.

11. The system of claim 1 , wherein a processor application corresponding to the first application processor checkpoint designates a processing scheme selected from a group of processing schemes comprising an at-least-once scheme and an exactly-once scheme.

12. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
receiving a first application processor checkpoint and an event expiration point, wherein the first application processor checkpoint corresponds to a first segment of an ordered event stream, wherein the ordered event stream further comprises a second segment, and wherein a second application processor checkpoint corresponds to the second segment; and
in response to determining that the first application processor checkpoint occurs at a lesser progress point than the event expiration point, blocking deletion of the event.

13. The non-transitory machine-readable medium of claim 12 , wherein the operations further comprise:
determining that the first application processor checkpoint occurs in a first epoch of the ordered event stream;
determining that the second application processor checkpoint occurs in a second epoch of the ordered event stream; and
determining that the first epoch occurs at a lesser progress point than the second epoch.

14. The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise determining that the first application processor checkpoint occurs at a lesser progress point that all other application processor checkpoints of the first epoch.

15. The non-transitory machine-readable medium of claim 12 , wherein the first application processor checkpoint is updatable, and wherein the event expiration point is updateable.

16. A method, comprising:
receiving, by a system comprising a processor, a first application processor checkpoint corresponding to a first segment of an ordered event stream, wherein the ordered event stream is stored via an ordered event stream storage system;
receiving, by the system, an event expiration point corresponding to the ordered event stream, and
in response to determining an expiration status of an event stored via the first segment of the ordered event stream based on the first application processor checkpoint and the event expiration point, causing, by the system, deletion of the event in the ordered event stream.

17. The method of claim 16 , wherein the determining the expiration status comprises determining, by the system, that the first application processor checkpoint occurs at a first progress point, that a second application processor checkpoint occurs at a second progress point, and that the first progress point is an earlier progress point than the second progress point.

18. The method of claim 17 , wherein the determining that the first progress point is an earlier progress point than the second progress point comprises determining, by the system, that a first epoch comprises the first progress point, that a second epoch comprises the second progress point, and that the first epoch occurs at an earlier progress point than the second epoch.

19. The method of claim 18 , further comprising determining, by the system, that the first progress point occurs at an earlier progress point that other progress points comprised in the first epoch.

20. The method of claim 16 , wherein the determining the expiration status comprises determining that the first application processor checkpoint occurs at a later progress point than the event expiration point, resulting in the expiration status being indicative of permitting deletion of the event.

Assignees

• Emc Ip Holding Co Llc

Inventors

• Danilov Mikhail
• Medvedev Igor
• Lei Lu
• Zhu Julius Jian

Classifications

• G06F11/1471 Primary

  involving logging of persistent data for recovery · CPC title

• G06F11/1407 Primary

  Checkpointing the instruction stream · CPC title

• G06F9/3555

  using scaling, e.g. multiplication of index · CPC title

• G06F9/542

  Event management; Broadcasting; Multicasting; Notifications · CPC title

• G06F11/3024

  where the computing system component is a central processing unit [CPU] · CPC title