Systems and methods for rule-based anomaly detection on IP network flow

Patent metadata

Field	Value
Publication number	US-9680877-B2
Application number	US-201514969591-A
Country	US
Kind code	B2
Filing date	Dec 15, 2015
Priority date	Dec 16, 2008
Publication date	Jun 13, 2017
Grant date	Jun 13, 2017

Abstract

A system to detect anomalies in internet protocol (IP) flows uses a set of machine-learning (ML) rules that can be applied in real time at the IP flow level. A communication network has a large number of routers equipped with flow monitoring capability. A flow collector collects flow data from the routers throughout the communication network and provides them to a flow classifier. At the same time, a limited number of locations in the network monitor data packets and generate alerts based on packet data properties. The packet alerts and the flow data are provided to a machine learning system that detects correlations between the packet-based alerts and the flow data to thereby generate a series of flow-level alerts. These rules are provided to the flow time classifier. Over time, the new packet alerts and flow data are used to provide updated rules generated by the machine learning system.

First claim

What is claimed is:

1. A system comprising:
a processor; and
memory comprising instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving a flow record of a data flow,
receiving a packet alert generated for a data packet, the packet alert generated based on a packet-level rule,
determining a time when the packet alert associated with the data packet was generated,
determining that the time when the packet alert associated with the data packet was generated occurred between a start time and an end time of the data flow,
in response to determining that the time when the packet alert associated with the data packet was generated occurred between the start time and the end time of the data flow, associating the packet alert with the data flow,
attributing, to the packet-level rule, a score in a form of a weighted sum Σ k w k p k (x) over flow level predicates p k (x), and
in response to the score exceeding a threshold, constructing a flow-level rule that, when applied to the data flow, generates an alarm associated with the data flow, wherein the flow-level rule identifies an anomaly in the data flow, and wherein a weight w k is chosen to minimize a classification error such that the flow-level rule, when applied to the data flow, generates the alarm associated with the data flow when the packet-level rule, when applied to the data packet, generates the packet alert for the data packet.

2. A method comprising:
receiving, by a system comprising a processor, a flow record of a data flow;
receiving, by the system, a packet alert generated for a data packet, the packet alert generated based on a packet-level rule,
determining, by the system, a time when the packet alert associated with the data packet was generated;
determining, by the system, that the time when the packet alert associated with the data packet was generated occurred between a start time and an end time of the data flow;
in response to determining that the time when the packet alert associated with the data packet was generated occurred between the start time and the end time of the data flow, associating, by the system, the packet alert with the data flow;
attributing, to the packet-level rule by the system, a score in a form of a weighted sum Σ k w k p k (x) over flow level predicates p k (x); and
in response to the score exceeding a threshold, constructing, by the system, a flow-level rule that, when applied to the data flow, generates an alarm associated with the data flow, wherein the flow-level rule identifies an anomaly in the data flow, and wherein a weight w k is chosen to minimize a classification error such that the flow-level rule, when applied to the data flow, generates the alarm associated with the data flow when the packet-level rule, when applied to the data packet, generates the packet alert for the data packet.

3. A memory storing instructions that, when executed by a processor of a system, cause the processor to perform operations comprising:
receiving a flow record of a data flow;
receiving a packet alert generated for a data packet, the packet alert generated based on a packet-level rule;
determining a time when the packet alert associated with the data packet was generated;
determining that the time when the packet alert associated with the data packet was generated occurred between a start time and an end time of the data flow;
in response to determining that the time when the packet alert associated with the data packet was generated occurred between the start time and the end time of the data flow, associating the packet alert with the data flow;
attributing, to the packet-level rule, a score in a form of a weighted sum Σ k w k p k (x) over flow level predicates p k (x); and
in response to the score exceeding a threshold, constructing a flow-level rule that, when applied to the data flow, generates an alarm associated with the data flow, wherein the flow-level rule identifies an anomaly in the data flow, and wherein a weight w k is chosen to minimize a classification error such that the flow-level rule, when applied to the data flow, generates the alarm associated with the data flow when the packet-level rule, when applied to the data packet, generates the packet alert for the data packet.

Assignees

• At & T Ip I Lp

Inventors

• Duffield Nicholas
• Haffner Patrick
• Krishnamurthy Balachander
• Ringberg Haakon Andreas

Classifications

• H04L41/00

  Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks · CPC title

• H04L47/10 Primary

  Flow control; Congestion control · CPC title

• G06N5/025

  Extracting rules from data · CPC title

• H04L41/16

  using machine learning or artificial intelligence · CPC title

• H04L43/16

  Threshold monitoring · CPC title