Monitoring containers running on container host devices for detection of anomalies in current container behavior

Patent metadata

Field	Value
Publication number	US-10936717-B1
Application number	US-201815883707-A
Country	US
Kind code	B1
Filing date	Jan 30, 2018
Priority date	Jan 30, 2018
Publication date	Mar 2, 2021
Grant date	Mar 2, 2021

Abstract

A method includes monitoring data of one or more containers running on one or more container host devices, a given one of the containers providing operating-system level virtualization for running at least one application. The method also includes determining a first set of behavior metrics for the given container based on the monitoring data, the first set of behavior metrics characterizing current behavior of the given container. The method further includes generating a model characterizing normal operation of the at least one application running in the given container using a second set of behavior metrics obtained during a learning period, utilizing the model to detect one or more anomalies in the first set of behavior metrics characterizing the current behavior of the given container, generating an alert responsive to detecting one or more anomalies in the first set of behavior metrics, and delivering the alert to a client device.

First claim

What is claimed is:

1. A method comprising:
monitoring data of one or more containers running on one or more container host devices, a given one of the containers is an execution structure configured to provide operating-system level virtualization for stand-alone execution of a single self-contained application in isolation from other single self-contained applications respectively executing on others of the one or more containers;
determining a first set of behavior metrics for the given container based on the monitoring data, the first set of behavior metrics characterizing behavior of the given container during execution of the single application;
generating a model characterizing normal operation of the single application running in the given container using a second set of behavior metrics obtained during a learning period;
utilizing the model to detect one or more anomalies in the first set of behavior metrics characterizing the behavior of the given container;
generating an alert responsive to detecting one or more anomalies in the first set of behavior metrics; and
sending the alert to a given client device over at least one network;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 wherein the first set of behavior metrics and the second set of behavior metrics comprise one or more system calls metrics, the system calls metrics characterizing sequences of assembly commands executed by the given container.

3. The method of claim 1 wherein generating the model characterizing normal operation of the single application running in the given container comprises:
grouping together sequences of N assembly commands in the second set of behavior metrics, wherein N is an integer greater than one; and
counting frequencies of occurrence of each of the sequences of N assembly commands in the second set of behavior metrics.

4. The method of claim 3 further comprising determining a set of normal sequences of assembly commands based on the frequencies of occurrence of the sequences of N assembly commands in the second set of behavior metrics, wherein utilizing the model to detect one or more anomalies in the first set of behavior metrics characterizing the behavior of the given container comprises:
grouping together sequences of N assembly commands in the first set of behavior metrics;
identifying a count of the number of sequences of N assembly commands in the first set of behavior metrics not in the set of normal sequences of assembly commands; and
detecting an anomaly when the count of the number of sequences of N assembly commands in the first set of behavior metrics not in the set of normal sequences of assembly commands exceeds a designated threshold.

5. The method of claim 3 wherein utilizing the model to detect one or more anomalies in the first set of behavior metrics characterizing the behavior of the given container comprises:
grouping together sequences of N assembly commands in the first set of behavior metrics;
identifying a frequency of occurrence of a given sequence of N assembly commands in the first set of behavior metrics;
comparing the frequency of occurrence of the given sequence of N assembly commands in the first set of behavior metrics with the frequency of occurrence of the given sequence of N assembly commands in the second set of behavior metrics; and
detecting an anomaly when a difference in the frequency of occurrence of the given sequence of N assembly commands in the first set of behavior metrics and the frequency of occurrence of the given sequence of N assembly commands in the second set of behavior metrics exceeds a designated threshold.

6. The method of claim 1 wherein the first set of behavior metrics and the second set of behavior metrics comprise one or more central processing unit (CPU) metrics, the CPU metrics comprising two or more of:
a number of elapsed enforcement period intervals for the given container;
a number of throttled period intervals for the given container;
a total time duration that the given container has been throttled;
a cumulative system CPU time consumed by the given container;
a cumulative system CPU time consumed per CPU by the given container; and
a cumulative user CPU time consumed by the given container.

7. The method of claim 1 wherein the first set of behavior metrics and the second set of behavior metrics comprise one or more memory metrics, the memory metrics comprising two or more of:
a number of byes of page cache memory utilized by the given container;
a number of memory usage limit hits incurred by the given container;
a cumulative count of memory allocation failures by the given container;
a size of the resident set size representing data belonging to processes on the given container that do not correspond to data on disk;
a container swap usage for the given container;
a memory usage for the given container; and
a working set for the given container, the working set representing pages that have been touched by a kernel within a designated time threshold.

8. The method of claim 1 wherein the first set of behavior metrics and the second set of behavior metrics comprise one or more file system metrics, the file system metrics comprising two or more of:
a number of available index nodes (inodes) for the given container;
a number of inodes allocated to the given container;
a number of input/output (IO) operations in progress for the given container;
a cumulative count of time spent performing IO operations for the given container;
an amount of data that can be consumed by the given container on the container file system;
a cumulative count of time spent reading and writing to the container file system;
a cumulative count of the amount of data read and written by the given container;
a cumulative count of reads and writes merged by the given container;
a cumulative count of reads and writes completed by the given container;
a cumulative count of sector reads completed by the given container; and
an amount of data consumed by the given container on the container file system.

9. The method of claim 1 wherein the first set of behavior metrics and the second set of behavior metrics comprise one or more networking metrics, the network metrics comprising two or more of:
a cumulative count of an amount of data received by the given container;
a cumulative count of errors encountered while the given container received data;
a cumulative count of packets dropped while the given container received data;
a cumulative count of an amount of data transmitted by the given container;
a cumulative count of errors encountered while the given container transmitted data;
a cumulative count of packets transmitted by the given container.

10. The method of claim 1 wherein utilizing the model to detect the one or more anomalies comprises at least one of the following:
using at least one of time-series outlier detection, k-nearest neighbor and a recurrent neural network; and
comparing the first set of behavior metrics against one or more threshold metric values, the threshold metric values being based on the second set of behavior metrics.

11. The method of claim 1 wherein the one or more threshold metric values comprises two or more threshold metric values for two or more distinct types of behavior metrics; and wherein the generated alert specifies which of the first set of behavior metrics have associated metric values exceeding the one or more threshold metric values.

12. The method of claim 1 further comprising modifying access by the given client device to the given co

Assignees

• Emc Ip Holding Co Llc

Inventors

• Herman Saffar Or
• Savir Amihai
• Golan Oron
• Ben Eliyahu Roie

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F9/5077

  Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455) · CPC title

• H04L63/1425

  Traffic logging, e.g. anomaly detection · CPC title

• G06F2009/45591

  Monitoring or debugging support · CPC title