Load distribution enabling detection of first appearance of a new property value in pipeline data processing

Patent metadata

Field	Value
Publication number	US-10867033-B2
Application number	US-201815928830-A
Country	US
Kind code	B2
Filing date	Mar 22, 2018
Priority date	Mar 22, 2018
Publication date	Dec 15, 2020
Grant date	Dec 15, 2020

Abstract

When processing events associated with a group comprising multiple different sub-groups, a hash function can be applied to the sub-group identifier to map the events associated with the sub-group to different computational elements used to process the group's events. The hash value can be a number between 1 and n or 0 and n−1 where n is the number of computational elements available to the group. Data concerning the last time a particular value for a property was encountered in an event stream can be retained. On each computational element assigned to the group, the detection of a particular property value in an event of a sub-group can be collected, periodically aggregated and sent to each of the computational elements used by the group, thereby enabling the first detection of a new property value within a group of events to be determined.

First claim

What is claimed:

1. A computing device comprising:
a processor; and
a memory connected to the processor,
the processor configured to execute instructions stored on the memory that cause the processor to:
generate hashes of identifiers associated with a plurality of sub-groups of a group, the hashes being used to divide events among a plurality of computational elements for processing;
extract appearance data representing when a particular new property value is utilized by at least two different computational elements at different times during processing of the events;
determine an initial appearance of the particular new property value according to the extracted appearance data; and
send a notification to at least one to the plurality of computational elements, the notification identifying the determined initial appearance of the particular new property value, a time associated with the determined initial appearance of the particular new property value, and the particular new property value, the notification being sent for synchronizing usage of the particular new property value across the plurality of computational elements.

2. The computing device of claim 1 , wherein all the events the same sub-group are processed by the same computational element.

3. The computing device of claim 1 , wherein the events of a plurality of sub-groups can be hashed to one computational element.

4. The computing device of claim 1 , wherein the initial appearance of the particular new property value represents the most recent appearance of the particular new property value within a triggering threshold time period since a previous appearance of the particular new property value.

5. The computer device of claim 4 , wherein the processor is further configured to execute instructions stored on the memory that cause the processor to:
send a report identifying the determined initial appearance of the particular new property value to a security center of the group.

6. The computing device of claim 4 , wherein the processor is further configured to execute instructions stored on the memory that cause the processor to:
retain a time of a last detection of the particular new property value to determine whether the triggering threshold time period is exceeded.

7. The computing device of claim 1 , wherein the group comprises a group of users sharing access to software in a multi-tenancy system.

8. The computing device of claim 1 , wherein state is synchronized between the plurality of sub-groups of the group by update events.

9. A method comprising:
receiving an event stream of events generated by a group, the group comprising sub-groups;
distributing the events among a plurality of computational elements based on hashes of sub-group identifiers such that different computational elements process different subsets of the events for different sub-groups;
obtaining appearance data representing when a particular new property value appears in a first subset of events processed by a first computational element and a second subset of events processed by a second computational element;
aggregating the appearance data to determine an initial appearance of the particular new property value in the first subset of events and the second subset of events;
generating a notification identifying the initial appearance of the particular new property value; and
providing the notification to at least one of the plurality of computational elements.

10. The method of claim 9 , further comprising:
performing the aggregating periodically.

11. The method of claim 10 , further comprising:
generating an alert indicating whether the initial appearance occurs in the first subset of events or the second subset of events.

12. The method of claim 10 , further comprising:
generating an alert identifying the particular new property value and a time of the initial appearance of the particular new property value.

13. The method of claim 9 , wherein the notification is sent to each of the plurality of computational elements.

14. A machine-readable medium storing instructions which, when executed by a machine, cause the machine to perform acts comprising:
receiving an event stream of events generated by a group, the group comprising sub-groups, wherein the events are distributed among a plurality of computational elements based on sub-group identifiers such that different computational elements process different subsets of the events for different sub-groups;
obtaining appearance data representing when a particular new property value appears in a first subset of events processed by a first computational element and a second subset of events processed by a second computational element;
determining an initial appearance of the particular new property value in the first subset of events and the second subset of events;
generating a notification identifying the initial appearance of the particular new property value; and
providing the notification to at least one of the plurality of computational elements.

15. The machine-readable medium of claim 14 , the acts further comprising:
performing the determining of the initial appearance repeatedly over time to identify additional initial appearances of the particular new property value separated by a configurable amount of time.

16. The machine-readable medium of claim 14 , the acts further comprising:
generating an alert indicating whether the initial appearance occurs in the first subset of events or the second subset of events.

17. The machine-readable medium of claim 14 , the acts further comprising:
generating an alert identifying the particular new property value and a time of the initial appearance of the particular new property value.

18. The machine-readable medium of claim 14 , wherein the notification is sent to each of the plurality of computational elements.

19. The machine-readable medium of claim 14 , the acts further comprising:
receiving the event stream from a particular tenant acting as an event source of the events.

20. The machine-readable medium of claim 14 , the sub-group identifiers comprising at least one of user identifiers, device identifiers, or network addresses.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Keren Shai
• Barav Ido
• Berkovitz Avihai
• Kaplan Shai

Classifications

• G06F11/3409 Primary

  for performance assessment · CPC title

• G06F16/137

  Hash-based (content-based indexing of textual data G06F16/31) · CPC title

• G06F21/55 Primary

  Detecting local intrusion or implementing counter-measures · CPC title

• G06F21/552

  involving long-term monitoring or reporting · CPC title

• G06F11/3072

  where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting · CPC title