Data stream processing system and methods for use therewith

Patent metadata

Field	Value
Publication number	US-12267378-B2
Application number	US-202418438776-A
Country	US
Kind code	B2
Filing date	Feb 12, 2024
Priority date	Dec 11, 2020
Publication date	Apr 1, 2025
Grant date	Apr 1, 2025

Abstract

A data stream processing system is operable to generate key assignment data indicating assignment of each of a plurality of subsets of a plurality of possible keys to a plurality of feed receiver modules by assigning one owner feed receiver module of the plurality of feed receiver modules to each key of the plurality of possible keys for a plurality of incoming messages, and by further assigning a set of follower feed receiver modules of the plurality of feed receiver modules to the each key of the plurality of possible keys. The plurality of incoming messages are processed via the plurality of feed receiver modules based on the key assignment data.

First claim

What is claimed is:

1. A method comprising:
generating key assignment data indicating assignment of each of a plurality of subsets of a plurality of possible keys to a plurality of feed receiver modules by assigning one owner feed receiver module of the plurality of feed receiver modules to each key of the plurality of possible keys for a plurality of incoming messages, and by further assigning a set of follower feed receiver modules of the plurality of feed receiver modules to the each key of the plurality of possible keys; and
processing the plurality of incoming messages via the plurality of feed receiver modules based on the key assignment data.

2. A data stream processing system comprises:
at least one processor; and
at least one memory that stores operational instructions that, when executed by the at least one processor, cause the data stream processing system to:
generate key assignment data indicating assignment of each of a plurality of subsets of a plurality of possible keys to a plurality of feed receiver modules by assigning one owner feed receiver module of the plurality of feed receiver modules to each key of the plurality of possible keys for a plurality of incoming messages, and by further assigning a set of follower feed receiver modules of the plurality of feed receiver modules to the each key of the plurality of possible keys; and
process the plurality of incoming messages via the plurality of feed receiver modules based on the key assignment data.

3. The data stream processing system of claim 2 , wherein at least one feed receiver module of the plurality of feed receiver modules maintains a queue in accordance with a time-based ordering based on timestamps of at least some of the plurality of incoming messages, and wherein the queue is maintained by the at least one feed receiver module to include only ones of the plurality of incoming messages of messages with timestamps that compare favorably to a time window.

4. The data stream processing system of claim 2 , wherein a proper subset of messages of the plurality of incoming messages includes followed ones of the plurality of incoming messages, based on having keys included in a followed subset of keys in a corresponding subset the plurality of possible keys, corresponding to keys in the corresponding subset to which at least one feed receiver module of the plurality of feed receiver modules is assigned as a follower in the key assignment data, and wherein the followed ones of the plurality of incoming messages are not emitted by the at least one feed receiver module.

5. The data stream processing system of claim 2 , wherein at least one feed receiver module in the plurality of feed receiver modules receives a proper subset of messages of the plurality of incoming messages, and wherein the operational instructions, when executed by the at least one processor, further cause the data stream processing system to:
determine key reassignment data indicating assignment of a new plurality of subsets of the plurality of possible keys to a proper subset of the plurality of feed receiver modules;
process the plurality of incoming messages via the plurality of feed receiver modules based on the key reassignment data by, via the at least one feed receiver module in the plurality of feed receiver modules:
receiving a second proper subset of messages of the plurality of incoming messages with keys included in a new subset of the new plurality of subsets of the plurality of possible keys assigned to at least one feed receiver module in the key reassignment data;
maintaining all messages of the second proper subset of messages that compare favorably to a time window; and
emitting only owned ones of the second proper subset of messages, based on having keys included in an owned subset of keys in the new subset, corresponding to ones of the new subset of the plurality of possible keys to which the at least one feed receiver module is assigned as an owner in the key reassignment data.

6. The data stream processing system of claim 5 , wherein a set difference between the owned subset of keys in the new subset and the owned subset of keys in a corresponding subset is non-null for at least one of the plurality of feed receiver modules.

7. The data stream processing system of claim 5 , wherein the operational instructions, when executed by the at least one processor, further cause the data stream processing system to, via at least one of the plurality of feed receiver modules assigned as an owner to at least one new key in the key reassignment data:
identify newly owned messages as a subset of the proper subset of messages, based on having keys included in a newly owned subset of keys to which the at least one feed receiver module is assigned as an owner in the key reassignment data; and
emit the newly owned messages.

8. The data stream processing system of claim 7 , wherein each of the proper subset of messages maintained by the at least one feed receiver module includes a corresponding timestamp, and wherein identifying the newly owned messages includes identifying ones of the proper subset of messages having a key included in the newly owned subset of keys and further having timestamps that are more-recent than another timestamp associated with the key for the key reassignment data.

9. The data stream processing system of claim 8 , wherein the another timestamp is generated based on at least one of: a timestamp of a most recently emitted message for the key by a prior owner for the key indicated in the key assignment data.

10. The data stream processing system of claim 5 , the operational instructions, when executed by the at least one processor, further cause the data stream processing system to:
determine to generate the key reassignment data based on detecting that one feed receiver in the plurality of feed receiver modules undergoes at least one of: a failure condition, or an offline condition; and
generate the key reassignment data based on assigning a new one of the plurality of feed receiver modules as the owner for each key in the owned subset of keys assigned to the one feed receiver in the key assignment data;
wherein the proper subset of the plurality of feed receiver modules is selected to not include the one feed receiver; and
wherein each new one of the plurality of feed receiver modules assigned as the owner for a corresponding key in the owned subset of keys assigned to the one feed receiver in the key assignment data is selected based on being assigned as a follower of the corresponding key in the key assignment data.

11. The data stream processing system of claim 10 , wherein a set of follower feed receiver modules of the plurality of feed receiver modules are assigned to the each key of the plurality of possible keys in the key assignment data based on:
assignment of a same, predefined number of feed receiver modules of the plurality of feed receiver modules to the each key of the plurality of possible keys; and
assignment of a set of follower rankings to the set of follower feed receiver modules;
wherein the each new one of the plurality of feed receiver modules is further selected based on having a most favorably ranked one of the set of follower rankings of the set of follower feed receiver modules assigned to the corresponding key in the key assignment data.

12. The data stream processing system of claim 11 , wherein a remaining subset of the set of follower feed receiver modules assigned to the corresponding key in the key assignment data are assigned new follower rankings in the key reassignment data based on preserving an original ordering of the set of follower rankings, and based on

Assignees

• Ocient Holdings LLC

Inventors

• Coombs Daniel

Classifications

• H04L65/75 Primary

  Media network packet handling · CPC title

• H04L65/756

  adapting media to device capabilities · CPC title

• G06F16/90335

  Query processing · CPC title

• G06F16/24568 Primary

  Data stream processing; Continuous queries · CPC title