Streaming anomaly detection

Patent metadata

Field	Value
Publication number	US-12348581-B2
Application number	US-201916682255-A
Country	US
Kind code	B2
Filing date	Nov 13, 2019
Priority date	Nov 13, 2019
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

Computational methods and systems to detect anomalous behaving resources and objects of a distributed computing system are described. Multiple streams of metric data representing usage of various resources of the distributed computing system are sent to a management system of the distributed computing system. The management system updates a performance model based on newly received metric values of the streams of metric data. The updated performance model is used to detect changes in one or more of the streams of metric data. The changes may be an indication of anomalous behavior at resources and objects associated with the streams of metric data. An anomaly listener is notified of anomalous behavior by the resource or object when a change in one or more of the streams of metric data is detected.

First claim

The invention claimed is:

1. A process stored in one or more data-storage devices and executed using one or more processors of a computer system to detect anomalous behavior exhibited by resources and objects of a distributed computing system, the process comprising:
receiving multiple streams of metric data generated by metric sources in the distributed computing system;
updating a performance model based on most recently received metric values of the streams of metric data, the performance being selected from a library of parametric, regression-based, and signal processing-based performance models;
performing anomaly detection to detect changes in one or more of the streams of metric data based on the updated performance model, the streams of metric data being assigned probability scores based on a probability density function; and
notifying an anomaly listener of anomalous behavior exhibited by a resource or object associated with a change detected in one or more of the streams of metric data.

2. The process of claim 1 wherein updating the performance model comprises:
for new metric values of the streams of metric data,
computing a mean of the recently received metric values, and
computing a sample standard deviation of the recently received metric values; and

for each new metric value of the streams of metric data, computing a standard-score model based on the recently received metric value, the mean, and the sample standard deviation.

3. The process of claim 1 wherein updating the performance model comprises:
computing a mean usage tuple from new metric values of the streams of metric data, each element of the mean-usage tuple corresponding to the mean usage of a resource of the distributed computing system;
forming a usage tuple from the new metric values of the resources;
computing a covariance matrix of the new metric values of the resources; and
computing a distance model that represents a distance from the usage tuple to the mean-usage tuple based on the usage tuple, the mean-usage tuple, and the covariance matrix.

4. The process of claim 1 wherein updating the performance model comprises:
for each stream of the multiple streams of metric data,
computing forecast metric values in a forecast interval; and
computing a forecast confidence intervals model for each of the forecast metric values.

5. The process of claim 1 wherein updating the performance model comprises:
for each stream of the streams of metric data,
determining if the stream of the metric data is a seasonal stream of metric data;
if the stream of metric data is a seasonal stream of metric data, computing a principal frequency of the stream of metric data based on new metric values in a current time window; and
computing an absolute difference between the principal frequency in the current time window and a principal frequency in a previous time window.

6. The process of claim 1 wherein performing anomaly detection to detect changes in one or more of the streams of metric data based on the updated performance model comprises:
determining a threshold based on the performance model; and
when the performance model violates the threshold, identifying the resource or object as exhibiting anomalous behavior.

7. A computer system to detect anomalous behavior exhibited by resources and objects of a distributed computing system, the system comprising:
one or more processors;
one or more data-storage devices; and
machine-readable instructions stored in the one or more data-storage devices that when executed using the one or more processors controls the system to perform operations comprising:
receiving multiple streams of metric data generated by metric sources in of the distributed computing system;
updating a performance model based on most recently received metric values of the streams of metric data, the performance being selected from a library of parametric, regression-based, and signal processing-based performance models;
performing anomaly detection to detect changes in one or more of the streams of metric data based on the updated performance model, the streams of metric data being assigned probability scores based on a probability density function; and
notifying an anomaly listener of anomalous behavior exhibited by a resource or object associated with a change detected in one or more of the streams of metric data.

8. The system of claim 7 wherein updating the performance model comprises:
for new metric values of the streams of metric data,
computing a mean of the recently received metric values, and
computing a sample standard deviation of the recently received metric values; and

for each new metric value of the streams of metric data, computing a standard-score model based on the recently received metric value, the mean, and the sample standard deviation.

9. The system of claim 7 wherein updating the performance model comprises:
computing a mean usage tuple from new metric values of the streams of metric data, each element of the mean-usage tuple corresponding to the mean usage of a resource of the distributed computing system
forming a usage tuple from the new metric values of the resources;
computing a covariance matrix of the new metric values of the resources; and
computing a distance model that represents a distance from the usage tuple to the mean-usage tuple based on the usage tuple, the mean-usage tuple, and the covariance matrix.

10. The system of claim 7 wherein updating the performance model comprises:
for each stream of the multiple streams of metric data,
computing forecast metric values in a forecast interval; and
computing a forecast confidence intervals model for each of the forecast metric values.

11. The system of claim 7 wherein updating the performance model comprises:
for each stream of the streams of metric data,
determining if the stream of the metric data is a seasonal stream of metric data;
if the stream of metric data is a seasonal stream of metric data, computing a principal frequency of the stream of metric data based on new metric values in a current time window; and
computing an absolute difference between the principal frequency in the current time window and a principal frequency in a previous time window.

12. The system of claim 7 wherein performing anomaly detection to detect changes in one or more of the streams of metric data based on the updated performance model comprises:
determining a threshold based on the performance model; and
when the performance model violates the threshold, identifying the resource or object as exhibiting anomalous behavior.

13. A non-transitory computer-readable medium encoded with machine-readable instructions that controls one or more processors of a computer system to perform the operations comprising:
receiving multiple streams of metric data generated by metric sources in a distributed computing system;
updating a performance model based on most recently received metric values of the streams of metric data, the performance being selected from a library of parametric, regression-based, and signal processing-based performance models;
performing anomaly detection to detect changes in one or more of the streams of metric data based on the updated performance model, the streams of metric data being assigned probability scores based on a probability density function; and
notifying an anomaly listener of anomalous behavior exhibited by a resource or object of the distributed computing system, the resource or object associated with a change detected in one or more of the streams of metric data.

14. The medium of

Assignees

• VMware LLC

Inventors

• Mathur Keshav
• Lu Jinyi
• Wang Xing
• Brown Darren
• Gao Peng
• Lin Junyuan
• Pedersen Paul

Classifications

• H04L43/08

  Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters · CPC title

• H04L41/147

  for predicting network behaviour · CPC title

• H04L41/40

  using virtualisation of network functions or resources, e.g. SDN or NFV entities · CPC title

• H04L41/145

  involving simulating, designing, planning or modelling of a network · CPC title

• H04L63/1425

  Traffic logging, e.g. anomaly detection · CPC title