Cross-stream transactions in a streaming data storage system

Patent metadata

Field	Value
Publication number	US-11593030-B2
Application number	US-202117316842-A
Country	US
Kind code	B2
Filing date	May 11, 2021
Priority date	May 11, 2021
Publication date	Feb 28, 2023
Grant date	Feb 28, 2023

Abstract

The described technology is generally directed towards cross-stream transactions in a streaming data storage system, which allows a writer application to commit multiple events to distinct data streams in a single transaction. The system creates a cross-stream transaction for a writer application, and the writer application adds events to the cross-stream transaction, indicating which destination data stream(s) each event's data is to be appended. The system adds the event to a subordinate transaction created for each specified data stream. Upon committing the cross-stream transaction, the system coordinates the committing of the subordinate transactions to their respective data streams. Alternatively, an application writer can request that an event be committed to multiple writer-specified data streams via a cross-stream micro-transaction, in which the streaming data storage system obtains the appropriate segment store instances and segments for each data stream, and coordinates the direct appending of the event to the appropriate segments.

First claim

What is claimed is:

1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
receiving a commit request to commit a cross-data stream transaction corresponding to a first data stream and a second data stream;
adding the first event data to a first subordinate transaction of the first data stream that is subordinate to the cross-data stream transaction;
adding the second event data to a second subordinate transaction of the second data stream;
committing the first subordinate transaction resulting in appending the first event data to the first data stream; and
committing the second subordinate transaction resulting in appending the second event data to the second data stream, wherein the second event data is different than the first event data.

2. The system of claim 1 , wherein the commit request comprises a cross-data stream micro-transaction request associated with a first identifier of the first data stream, a second identifier of the second data stream and common event data comprising the first event data and the second event data.

3. The system of claim 2 , wherein the operations further comprise communicating with a first controller associated with the first data stream to determine a first destination segment and a first segment store usable to append the first event data, and communicating with a second controller associated with the second data stream to determine a second destination segment and a second segment store usable to append the second event data.

4. The system of claim 1 , wherein the operations further comprise receiving a request corresponding to creating the cross-data stream transaction, and, in response to the request, creating the cross-data stream transaction comprising creating a cross-data stream transaction data structure comprising a cross-data stream transaction identifier, and returning the cross-data stream transaction identifier.

5. The system of claim 1 , wherein the operations further comprise receiving the first event data in association with the cross-data stream transaction identifier and in association with a data stream identifier dataset comprising the first identifier of the first data stream.

6. The system of claim 1 , wherein the operations further comprise receiving the first event data in association with the cross-data stream transaction identifier and in association with a data stream identifier dataset comprising the first identifier of the first data stream and a third identifier of a third data stream.

7. The system of claim 1 , wherein the operations further comprise creating the first subordinate transaction of the first data stream.

8. The system of claim 7 , wherein the creating the first subordinate transaction of the first data stream comprises receiving the first event data, and, in response to receiving the first event data, determining that the first subordinate transaction does not already exist.

9. A method, comprising:
coordinating, via a streaming data storage system comprising a processor, a cross-data stream transaction, the coordinating comprising:
receiving a cross-data stream transaction commit request, and
in response to receiving the cross-data stream transaction commit request, committing the cross-data stream transaction comprising communicating with a first controller associated with a first data stream, and communicating with a second controller associated with a second data stream, wherein the first controller of the first data stream manages a first subordinate transaction to which first event data is added, the first subordinate transaction being subordinate to the cross-data stream transaction, wherein the second controller of a second data stream manages a second subordinate transaction to which second event data is added, the second subordinate transaction being subordinate to the cross-data stream transaction, wherein the committing the cross-data stream transaction comprises instructing the first controller to commit the first subordinate transaction and instructing the first controller to commit the second subordinate transaction resulting in appending first event data to the first data stream in conjunction with appending second event data to the second data stream as part of the committing the cross-data stream transaction, and wherein the second event data is different than the first event data.

10. The method of claim 9 , further comprising receiving a request to create the cross-data stream transaction, and, in response to the request, creating the cross-data stream transaction comprising creating a cross-data stream transaction data structure having a cross-data stream transaction identifier, and returning the cross-data stream transaction identifier.

11. The method of claim 9 , further comprising instructing the first controller to create the first subordinate transaction and instructing the second controller to create the second subordinate transaction.

12. The method of claim 11 , further comprising receiving the first event data in association with an identifier of the first data stream, and, in response to receiving the first event data, adding the first event data to the first subordinate transaction.

13. The method of claim 9 , further comprising, in response to receiving the first event data, determining whether the first subordinate transaction already exists, and, in response to determining that the first subordinate transaction does not already exist, instructing the first controller to create the first subordinate transaction.

14. The method of claim 9 , wherein the first event data and the second event data are common event data, and wherein the receiving the cross-data stream transaction commit request comprises receiving a cross-data stream micro-transaction request associated with a first identifier of the first data stream and a second identifier of the second data stream.

15. The method of claim 14 , further comprising communicating with the first controller to determine a first destination segment and a first segment store to append the common event data as the first event data, and communicating with the second controller to determine a second destination segment and a second segment store to append the common event data as the second event data.

16. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a streaming data storage system, facilitate performance of operations, the operations comprising:
controlling a cross-data stream transaction, comprising creating a cross-data stream transaction data structure and associating the cross-data stream transaction with a cross-data stream transaction identifier;
receiving a first event associated with the cross-data stream transaction identifier and a first identifier of a first data stream;
adding first event data of the first event to a first subordinate transaction data structure of a first subordinate transaction associated with the first data stream, the first subordinate transaction being subordinate to the cross-data stream transaction;
receiving a second event associated with the cross-data stream transaction identifier and a second identifier of a second data stream;
adding second event data of second first event to a second subordinate transaction data structure of a second subordinate transaction associated with the second data stream, the second subordinate transaction being subordinate to the cross-data stream transaction;
receiving a

Assignees

• Emc Ip Holding Co Llc

Inventors

• Danilov Mikhail
• Altaye Yohannes

Classifications

• G06F3/0673

  Single storage device · CPC title

• G06F3/0659 Primary

  Command handling arrangements, e.g. command buffers, queues, command scheduling · CPC title

• G06F3/0635

  by changing the path, e.g. traffic rerouting, path reconfiguration · CPC title

• G06F3/0683 Primary

  Plurality of storage devices · CPC title

• G06F3/0613

  in relation to throughput · CPC title