Complex event processing of computer network data

Patent metadata

Field	Value
Publication number	US-9667641-B2
Application number	US-201514929042-A
Country	US
Kind code	B2
Filing date	Oct 30, 2015
Priority date	Aug 31, 2015
Publication date	May 30, 2017
Grant date	May 30, 2017

Abstract

A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.

First claim

What is claimed is:

1. A method comprising:
transforming in real time raw event data, representing a sequence of events and captured from multiple sources in a computer network, into a stream of event feature sets without a known end-point to the stream, where the raw event data includes time-stamped machine data;
computing in real-time a score by processing a time slice of the stream of event feature sets through an active version of a machine learning model, wherein the time slice includes a most recent event feature set in the stream of event feature sets;
training, in parallel with said processing the time slice and responsive in real-time to said transforming the raw event data, a non-active version of the machine learning model with the time slice that is being processed through the active version for scoring, wherein the machine learning model is trained to represent a particular entity involved in a computer network activity represented by the raw event data;
identifying, by comparing the score against a threshold, a security-related anomaly or a security-related threat to enable remediation of the security-related anomaly or the security-related threat in the computer network as the stream of event feature sets is processed in real-time;
determining that the non-active version of the machine learning model is ready for active deployment based on at least one of: a number of event feature sets that have been used to train the non-active version, length of time that the non-active version has been in training, or whether a model state of the non-active version is converging; and
live-swapping in the non-active version as the active version to compute another score by processing a subsequent time slice from the stream of event feature sets through the live-swapped-in active version of the machine learning model.

2. The method of claim 1 , wherein the time slice is the most recent time slice received from the stream of event feature sets.

3. The method of claim 1 , wherein the time slice corresponds to an event in the sequence of events; and wherein said identifying the security-related anomaly or the security-related threat includes determining that the security-related anomaly or the security-related threat corresponds to the event.

4. The method of claim 1 , further comprising training the machine learning model by processing a previous time slice of the stream of event feature sets, prior to said computing the score.

5. The method of claim 1 , wherein said processing the time slice through the active version of the machine learning model includes processing the time slice through a model deliberation process logic configured by the active version of the machine learning model; wherein said training the non-active version of the machine learning model includes processing the time slice through a model training process logic; and wherein the model training process logic and the model deliberation process logic correspond to a particular machine learning model type.

6. The method of claim 1 ,
wherein
said live swapping includes re-configuring, without first terminating, a model deliberation process thread that is performing said computing in real-time.

7. The method of claim 1 , wherein the machine learning model is an unsupervised machine learning model.

8. The method of claim 1 , furthering comprising processing the stream of event feature sets through a plurality of machine learning models of different types to detect security-related anomalies or threats of different types.

9. The method of claim 1 , wherein the machine learning model is a supervised machine learning model or a semi-supervised machine learning model.

10. The method of claim 1 , wherein the machine learning model is a deep machine learning model.

11. The method of claim 1 , wherein the machine learning model is specific to a particular entity involved in the sequence of events, wherein the entity is a user, a device, a system, a network resource locator, an application, a process thread, or any combination thereof.

12. The method of claim 1 , wherein the machine learning model performs behavioral analysis of a particular entity involved in the sequence of events.

13. The method of claim 1 , wherein the machine learning model performs peer group analysis amongst entities involved in the sequence of events.

14. The method of claim 1 , wherein the machine learning model performs time series analysis on a sequence of the event feature sets.

15. The method of claim 1 , wherein the machine learning model performs graph correlation analysis amongst entities involved in the sequence of events.

16. The method of claim 1 , wherein the machine learning model includes a state machine specific to an entity involved in an event represented in the time slice.

17. The method of claim 1 , wherein said computing the score includes processing the stream of event feature sets through the machine learning model according to model deliberation processing logic specified by a model type associated with the machine learning model.

18. The method of claim 1 , further comprising training the machine learning model according to model training processing logic specified by a model type associated with the machine learning model.

19. The method of claim 1 , further comprising incrementally updating the machine learning model in real-time using the time slice of the event feature sets, wherein said incremental updating includes:
isolating a portion of a model state representative of the machine learning model affected by the event feature sets; and
re-training only the portion of the model state.

20. The method of claim 1 , further comprising simultaneously training multiple machine learning models, associated with different purposes or different entities, in real-time.

21. The method of claim 1 , wherein computing the score is performed in a distributed computation system implementing a task-parallel distributed data processing engine.

22. The method of claim 1 , further comprising training the machine learning model in a distributed computation system implementing a task-parallel distributed data processing engine.

23. The method of claim 1 , further comprising training the machine learning model in real-time using single-pass training processing logic.

24. The method of claim 1 , further comprising:
training the machine learning model; and
storing a model state of the machine learning model, resulting from said training, in a distributed cache for use in said computing of the score.

25. The method of claim 1 , further comprising outputting, to a user via a computer output device, a representation of the security-related anomaly or the security-related threat, in response to determining the security-related anomaly or the security-related threat.

26. The method of claim 1 , wherein the threshold is defined by the machine learning model.

27. A system comprising:
at least one hardware processor implementing an extract transform load (ETL) engine configured to transform in real time raw event data, representing a sequence of events and captured from multiple sources in a computer network, into an stream of event feature sets without a known end-point to the stream, where the raw event data includes time-stamped machine data;
at least one hardware processor implementing a model execution engine configured to compute in real-t

Assignees

• Splunk Inc

Inventors

• Muddu Sudhakar
• Tryfonas Christos

Classifications

• G06N20/20 Primary

  Ensemble learning · CPC title

• H04L63/1416

  Event detection, e.g. attack signature detection · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F40/134

  Hyperlinking · CPC title

• H04L63/1425 Primary

  Traffic logging, e.g. anomaly detection · CPC title