Virtual partitioning of a shared message bus

Patent metadata

Field	Value
Publication number	US-11620164-B1
Application number	US-202017033253-A
Country	US
Kind code	B1
Filing date	Sep 25, 2020
Priority date	Sep 25, 2020
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

According to embodiments, a method for virtual partitioning of data includes receiving a data stream comprising a plurality of traces, each trace comprising a plurality of spans from a plurality of users. The method also includes assigning the plurality of traces of the data stream to a plurality of virtual partitions based on each user of the plurality of users, each virtual partition of the plurality of virtual partitions comprising data of a user of the plurality of users. The method also includes scheduling at least a subset of the plurality of virtual partitions to at least one user partition of a shared topic, the at least one user partition comprising data from at least one virtual partition of at least one user of the plurality of users. The method also includes indexing each user partition of the shared topic based on each user and each virtual partition.

First claim

What is claimed is:

1. A computer-implemented method for virtual partitioning of data, the method comprising:
receiving a data stream comprising a plurality of traces, each of the plurality of traces comprising a plurality of spans from a plurality of users;
assigning the plurality of traces of the data stream to a plurality of virtual partitions based on each user of the plurality of users, each virtual partition of the plurality of virtual partitions comprising data of a user of the plurality of users;
scheduling at least a subset of the plurality of virtual partitions to at least one user partition of a shared topic, the at least one user partition comprising data from at least one virtual partition of at least one user of the plurality of users; and
indexing each of the at least one user partition of the shared topic based on each of the plurality of users and each of the plurality of virtual partitions, wherein the indexing comprises processing each user partition through a consumer process, wherein each consumer process is based on a publication-subscription service partition.

2. The computer-implemented method of claim 1 , wherein scheduling the subset of the plurality of virtual partitions is based at least in part on a weighted stable load balancing algorithm.

3. The computer-implemented method of claim 1 , wherein each of the at least one user partitions comprises a maximum capacity based on a maximum throughput of a consumer.

4. The computer-implemented method of claim 1 , wherein a number of virtual partitions for each user is based on a total trace load and a trace load capacity of each of the plurality of virtual partitions.

5. The computer-implemented method of claim 1 , further comprising:
assigning each of the plurality of virtual partitions to a least loaded user partition based on at least one of a trace load, a cardinality, and a partition identifier.

6. The computer-implemented method of claim 1 , further comprising:
storing schedules of the shared topic in a database.

7. The computer-implemented method of claim 1 , further comprising:
adding a new virtual partition to a schedule of the shared topic.

8. The computer-implemented method of claim 1 , further comprising:
adding a new user partition to a schedule of the shared topic.

9. The computer-implemented method of claim 1 , further comprising:
increasing at least one of a cardinality or a trace load of a schedule of the shared topic.

10. A system for virtual partitioning of data, the system comprising:
at least one memory having instructions stored thereon; and
at least one processor configured to execute the instructions, wherein the at least one processor is configured to:
receive a data stream comprising a plurality of traces, each of the plurality of traces comprising a plurality of spans from a plurality of users;
assign the plurality of traces of the data stream to a plurality of virtual partitions based on each user of the plurality of users, each virtual partition of the plurality of virtual partitions comprising data of a user of the plurality of users;
schedule at least a subset of the plurality of virtual partitions to at least one user partition of a shared topic, the at least one user partition comprising data from at least one virtual partition of at least one user of the plurality of users; and
index each of the at least one user partitions of the shared topic based on each of the plurality of users and each virtual partition wherein the processor is configured to process each of the at least one user partitions through a consumer process, wherein each consumer process is based on a publication-subscription service partition.

11. The system of claim 10 , wherein scheduling the subset of the plurality of virtual partitions is based at least in part on a weighted stable load balancing algorithm.

12. The system of claim 10 , wherein each of the at least one user partitions comprises a maximum capacity based on a maximum throughput of a consumer.

13. The system of claim 10 , wherein a number of virtual partitions for each of the plurality of users is based on a total trace load and a trace load capacity of each virtual partition.

14. The system of claim 10 , wherein the processor is further configured to:
assign each of the plurality of virtual partitions to a least loaded user partition based on at least one of a trace load, a cardinality, and a partition identifier.

15. The system of claim 10 , wherein the processor is further configured to:
store schedules of the shared topic in a database.

16. The system of claim 10 , wherein the processor is further configured to:
add a new virtual partition to a schedule of the shared topic.

17. The system of claim 10 , wherein the processor is further configured to:
add a new user partition to a schedule of the shared topic.

18. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations for virtual partitioning of data, the operations comprising:
receiving a data stream comprising a plurality of traces, each of the plurality of traces comprising a plurality of spans from a plurality of users;
assigning the plurality of traces of the data stream to a plurality of virtual partitions based on each user of the plurality of users, each virtual partition of the plurality of virtual partitions comprising data of a user of the plurality of users;
scheduling at least a subset of the plurality of virtual partitions to at least one user partition of a shared topic, the at least one user partition comprising data from at least one virtual partition of at least one user of the plurality of users; and
indexing each of the at least one user partitions of the shared topic based on each user and each of the plurality of virtual partitions, wherein the indexing comprises processing each user partition through a consumer process, wherein each consumer process is based on a publication-subscription service partition.

Assignees

• Splunk Inc

Inventors

• Karis Steven
• Petazzoni Maxime
• Pound Matthew William
• Smith Charles
• Yang Chengyu

Classifications

• G06F9/542 Primary

  Event management; Broadcasting; Multicasting; Notifications · CPC title

• G06F9/4881

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F11/3466

  Performance evaluation by tracing or monitoring · CPC title

• G06F11/3636

  by tracing the execution of the program · CPC title

• G06F11/364

  tracing values on a bus · CPC title