Stream browser for data streams

Patent metadata

Field	Value
Publication number	US-11599546-B2
Application number	US-202016864905-A
Country	US
Kind code	B2
Filing date	May 1, 2020
Priority date	May 1, 2020
Publication date	Mar 7, 2023
Grant date	Mar 7, 2023

Abstract

The described technology is generally directed towards a stream browser that provides a human-readable (e.g., hierarchical) view of event data for a data stream of a data streaming storage service. A stream browser associated with a data stream maintains routing key and event data for the data stream, e.g., in a primary index. The routing keys can correspond to container structures, e.g., folders and subfolders, with the events associated with each routing key corresponding to hierarchical children of the routing key parent container structures (e.g., like files). A secondary index can be used to evict the oldest events from the primary index to limit the number of data stream events returnable by the stream browser to a practical number.

First claim

What is claimed is:

1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
receiving a request to return a hierarchical view on stream data of a data stream, the request comprising routing key information, wherein the data stream preserves stream data event order based on time according to a flat temporal storage topology, wherein the stream data is segregated into one or more segments of the data stream according to a routing key, and wherein a data structure comprises a routing key corresponding to non-temporal event data of an event of the events being stored via the data stream;
in response to the request, reading the data structure to obtain the non-temporal event data corresponding to the routing key based on the routing key information; and
rendering a display of the hierarchical view of the data stream without perturbing the data event order of the underlying data stream, the hierarchical view being based on the routing key corresponding to the non-temporal event data rather than rendering a flat view based on the flat temporal topology of the data stream, wherein the hierarchical view represents the routing key as a container structure, and wherein the container structure is a hierarchical parent containing the event data associated with the container structure.

2. The system of claim 1 , wherein the receiving the request comprises receiving a list folder content request, wherein the routing key information in the request specifies a first part of the routing key that corresponds to a folder, and wherein the returning the hierarchical view comprises listing respective one or more second parts of the routing key as respective one or more subfolders with respective one or more events as the event data associated with the respective one or more subfolders.

3. The system of claim 1 , wherein the receiving the request comprises receiving a list folder content request, wherein the routing key information in the request specifies null or an empty string, and wherein the returning the hierarchical view comprises listing subfolder content and event content of a root folder.

4. The system of claim 1 , wherein the receiving the request comprises receiving a read event request, wherein the routing key information in the request specifies a full routing key, and wherein the returning the hierarchical view comprises listing one or more events associated with the full routing key as the event data associated with the container structure corresponding to the full routing key.

5. The system of claim 1 , wherein the event data comprises two or more events, wherein the data structure comprises stream position data for each of the two or more events, and wherein the operations further comprise sorting the events by respective stream position data before returning the hierarchical view of the data stream.

6. The system of claim 1 , wherein the data structure is a first data structure, and wherein the operations further comprise maintaining a second data structure that relates respective routing keys to respective event positions in the data stream, wherein the respective event positions in the data stream are embodied in the flat temporal topology as a result of the events having been sequentially written into the data stream as a function of time, wherein the respective event positions for a respective routing key identify a respective first position in the data stream of a least recent event associated with the respective routing key and a respective second position in the data stream of a most recent event associated with the respective routing key.

7. The system of claim 6 , wherein the event data comprises two or more events, and wherein the operations further comprise accessing the second data structure based on the routing key to evict one or more least recent events from the first data structure.

8. The system of claim 6 , wherein the data stream is associated with a stream cut, wherein the data stream is truncated to remove events based on the stream cut, and wherein the operations further comprise, accessing the second data structure based on the stream cut to remove event data from the first data structure that correspond to truncated events, and to modify event position data in the second data structure.

9. The system of claim 1 , wherein the event data exceeds a resource limitation, and wherein the returning the hierarchical view of the data stream comprises providing a sliding window of event data that does not exceed the resource limitation.

10. The system of claim 1 , wherein the maintaining the data structure comprising the routing keys for the data stream comprises maintaining a hierarchical data index.

11. A method comprising:
logically associating a stream browser with a data stream of events comprising routing keys, wherein events that have been written into the data stream are stored in an order that both preserves data event order by time according to a flat temporal storage topology and separates data events into one or more data storage segments of the data stream according to the routing keys, the stream browser maintaining a data structure comprising the routing keys according to a hierarchical topology based on the values of the routing keys rather than the sequential order of the events in the data stream;
receiving a request at the stream browser to return a hierarchical view on stored data of the data stream, the request comprising routing key information;
in response to the request, accessing the data structure to obtain a routing key corresponding to the routing key information and events associated with the routing key; and
presenting, via a display device, a hierarchical view of the data stream without perturbing the data event order of the data stream, comprising representing the routing key as a folder that is a hierarchical parent containing the events associated with the routing key.

12. The method of claim 11 , wherein the receiving the request comprises receiving a list folder content request, wherein the routing key information in the request specifies a first part of the routing key that corresponds to a folder, and wherein the returning the hierarchical view comprises listing respective one or more second parts of the routing key as respective one or more subfolders with respective one or more events associated with the respective one or more subfolders.

13. The method of claim 11 , wherein the receiving the request comprises receiving a read event request, wherein the routing key information in the request specifies a full routing key, and wherein the returning the hierarchical view comprises listing the full routing key as the folder containing the events.

14. The method of claim 11 , wherein the data structure is a first data structure, and wherein the stream browser further maintains a second data structure that relates the routing key to a first stream position of a least recent event associated with the routing key and a second stream position of a most recent event associated with the routing key.

15. The method of claim 14 , further comprising, evicting events from the first data structure based on the least recent event.

16. The method of claim 11 , further comprising, configuring the hierarchical view of the data stream as an interactive sliding window over a selectable portion of the events associated with the routing key.

17. A non-transitory machine-readable storage medium, comprising executable instructions t

Assignees

• Emc Ip Holding Co Llc

Inventors

• Danilov Mikhail
• Medvedev Igor

Classifications

• G06F16/1734

  Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs · CPC title

• G06F16/2228

  Indexing structures · CPC title

• G06F16/13 Primary

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title